Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 20:51:20.188496 416 test_util.cc:274] Using random seed: 2104377892
2025-01-12T20:51:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:20Z Disabled control of system clock
I20250112 20:51:20.221149 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35199
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 20:51:20.538828 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:20.539101 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:20.539176 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:20.546398 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:20.546512 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:20.546553 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:20.546605 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:20.555859 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:20.557337 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:20.558892 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:20.569850 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:20.570737 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:20.571663 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:20.572521 424 server_base.cc:1034] running on GCE node
I20250112 20:51:20.573356 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:20.574735 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:20.575948 424 hybrid_clock.cc:648] HybridClock initialized: now 1736715080575872 us; error 61 us; skew 500 ppm
I20250112 20:51:20.576243 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:20.582336 424 webserver.cc:458] Webserver started at http://127.0.104.62:40001/ using document root <none> and password file <none>
I20250112 20:51:20.583053 424 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:20.583173 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:20.583449 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:20.586089 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data/instance:
uuid: "b04ae76622ac46f5a584a582e372bc05"
format_stamp: "Formatted at 2025-01-12 20:51:20 on dist-test-slave-4dz2"
I20250112 20:51:20.586789 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal/instance:
uuid: "b04ae76622ac46f5a584a582e372bc05"
format_stamp: "Formatted at 2025-01-12 20:51:20 on dist-test-slave-4dz2"
I20250112 20:51:20.591771 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250112 20:51:20.594988 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:20.596205 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:20.596349 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal
uuid: "b04ae76622ac46f5a584a582e372bc05"
format_stamp: "Formatted at 2025-01-12 20:51:20 on dist-test-slave-4dz2"
I20250112 20:51:20.596516 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:20.623329 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:20.624126 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:20.624397 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:20.643143 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35199
I20250112 20:51:20.643172 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35199 every 8 connection(s)
I20250112 20:51:20.644459 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:20.646150 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 424
I20250112 20:51:20.646451 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:20.649297 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:20.650225 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 20:51:20.656930 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:20.673878 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
W20250112 20:51:20.675004 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
W20250112 20:51:20.676400 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
I20250112 20:51:20.725699 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } attempt: 1
W20250112 20:51:20.727456 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
I20250112 20:51:20.795827 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } attempt: 2
W20250112 20:51:20.797590 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
I20250112 20:51:20.888877 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } attempt: 3
W20250112 20:51:20.890682 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
W20250112 20:51:20.987385 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:20.987725 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:20.987855 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:20.995828 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:20.995975 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:20.996049 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:20.996131 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:21.005316 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:21.006970 492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:21.008610 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:21.019224 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:21.020164 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:21.021031 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:21.021885 492 server_base.cc:1034] running on GCE node
I20250112 20:51:21.022607 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:21.023907 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:21.025125 492 hybrid_clock.cc:648] HybridClock initialized: now 1736715081025045 us; error 45 us; skew 500 ppm
I20250112 20:51:21.025419 492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:21.027521 492 webserver.cc:458] Webserver started at http://127.0.104.61:39277/ using document root <none> and password file <none>
I20250112 20:51:21.028699 492 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:21.028836 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:21.029124 492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:21.031586 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/instance:
uuid: "b985fd656ce44131bc91ccfeb79587c5"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.032307 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal/instance:
uuid: "b985fd656ce44131bc91ccfeb79587c5"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.033919 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } attempt: 4
W20250112 20:51:21.035632 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111)
I20250112 20:51:21.037559 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250112 20:51:21.040588 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:21.041837 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250112 20:51:21.041970 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
uuid: "b985fd656ce44131bc91ccfeb79587c5"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.042116 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:21.074290 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:21.075086 492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:21.075333 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:21.093793 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39271
I20250112 20:51:21.093825 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39271 every 8 connection(s)
I20250112 20:51:21.095181 492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
I20250112 20:51:21.098294 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 492
I20250112 20:51:21.098444 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal/instance
I20250112 20:51:21.098722 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:21.099457 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:45153
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 20:51:21.104898 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.118336 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.126804 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
W20250112 20:51:21.127727 513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
W20250112 20:51:21.129091 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.178400 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 1
W20250112 20:51:21.180250 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.248479 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 2
W20250112 20:51:21.250211 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.334964 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } attempt: 5
I20250112 20:51:21.341259 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.341410 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 3
W20250112 20:51:21.342948 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
W20250112 20:51:21.343402 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.394176 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 1
W20250112 20:51:21.395922 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
W20250112 20:51:21.424064 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:21.424331 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:21.424402 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:21.432246 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:21.432368 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:21.432410 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:21.432456 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:21.440809 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:21.442271 563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:21.443883 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:21.454936 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:21.455520 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:21.456440 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:21.457298 563 server_base.cc:1034] running on GCE node
I20250112 20:51:21.457996 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:21.459271 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:21.460485 563 hybrid_clock.cc:648] HybridClock initialized: now 1736715081460434 us; error 31 us; skew 500 ppm
I20250112 20:51:21.460819 563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:21.462858 563 webserver.cc:458] Webserver started at http://127.0.104.60:44167/ using document root <none> and password file <none>
I20250112 20:51:21.464010 563 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:21.464135 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:21.464118 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 2
I20250112 20:51:21.464597 563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20250112 20:51:21.465827 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.467388 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/instance:
uuid: "0f5c09e899184de6b770dca873501062"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.468117 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal/instance:
uuid: "0f5c09e899184de6b770dca873501062"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.472975 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250112 20:51:21.476207 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:21.477449 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:21.477578 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
uuid: "0f5c09e899184de6b770dca873501062"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:21.477725 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:21.486609 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 4
W20250112 20:51:21.488200 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111)
I20250112 20:51:21.498051 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:21.498757 563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:21.499002 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:21.517550 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:45153
I20250112 20:51:21.517570 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:45153 every 8 connection(s)
I20250112 20:51:21.518782 563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
I20250112 20:51:21.522159 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:21.526432 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 563
I20250112 20:51:21.526590 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal/instance
I20250112 20:51:21.528182 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.540601 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.547034 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20250112 20:51:21.560519 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Bootstrap starting.
I20250112 20:51:21.563680 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:21.564648 632 log.cc:826] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:21.566784 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: No bootstrap required, opened a new log
I20250112 20:51:21.572117 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 3
I20250112 20:51:21.573596 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.573982 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:21.574121 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5c09e899184de6b770dca873501062, State: Initialized, Role: FOLLOWER
I20250112 20:51:21.574744 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.575592 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.575888 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:21.576609 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:21.583743 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Bootstrap starting.
W20250112 20:51:21.587466 652 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 20:51:21.587694 652 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 20:51:21.588521 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:21.589534 490 log.cc:826] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:21.591620 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: No bootstrap required, opened a new log
I20250112 20:51:21.584280 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:35199: OK, instance_id { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" instance_seqno: 1736715080637667 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 20:51:21.593168 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 20:51:21.598318 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.598579 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:21.598691 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04ae76622ac46f5a584a582e372bc05, State: Initialized, Role: FOLLOWER
I20250112 20:51:21.599212 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.599999 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.600317 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:21.600888 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 20:51:21.612553 666 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 20:51:21.612733 666 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 20:51:21.610795 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:39271: OK, instance_id { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" instance_seqno: 1736715081089089 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 20:51:21.619289 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 20:51:21.627230 444 tablet.cc:2367] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:21.750845 586 tablet.cc:2367] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:21.787428 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } attempt: 5
I20250112 20:51:21.799281 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Bootstrap starting.
I20250112 20:51:21.804567 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:21.805830 561 log.cc:826] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:21.808697 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: No bootstrap required, opened a new log
I20250112 20:51:21.818218 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.818567 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:21.818701 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b985fd656ce44131bc91ccfeb79587c5, State: Initialized, Role: FOLLOWER
I20250112 20:51:21.819375 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.820297 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.820670 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:21.821523 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 20:51:21.827687 515 tablet.cc:2367] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:21.840642 492 master_runner.cc:263] No leader master found from master b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:21.840953 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 20:51:21.843722 680 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 20:51:21.843845 680 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 20:51:21.867048 655 raft_consensus.cc:491] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:21.867328 655 raft_consensus.cc:513] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.869321 655 leader_election.cc:290] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271), 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153)
I20250112 20:51:21.870616 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b04ae76622ac46f5a584a582e372bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5c09e899184de6b770dca873501062" is_pre_election: true
I20250112 20:51:21.871148 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b04ae76622ac46f5a584a582e372bc05 in term 0.
I20250112 20:51:21.872030 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b04ae76622ac46f5a584a582e372bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:21.872491 536 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b04ae76622ac46f5a584a582e372bc05 in term 0.
I20250112 20:51:21.873361 442 leader_election.cc:304] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5; no voters:
I20250112 20:51:21.878382 655 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:21.878544 655 raft_consensus.cc:491] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:21.878650 655 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:21.881814 655 raft_consensus.cc:513] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.883034 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b04ae76622ac46f5a584a582e372bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5"
I20250112 20:51:21.883230 536 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:21.886636 536 raft_consensus.cc:2463] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b04ae76622ac46f5a584a582e372bc05 in term 1.
I20250112 20:51:21.887125 442 leader_election.cc:304] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5; no voters:
I20250112 20:51:21.888653 655 leader_election.cc:290] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [CANDIDATE]: Term 1 election: Requested vote from peers b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271), 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153)
I20250112 20:51:21.889720 655 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:21.890722 655 raft_consensus.cc:695] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [term 1 LEADER]: Becoming Leader. State: Replica: b04ae76622ac46f5a584a582e372bc05, State: Running, Role: LEADER
I20250112 20:51:21.891542 655 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:21.904959 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b04ae76622ac46f5a584a582e372bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5c09e899184de6b770dca873501062"
I20250112 20:51:21.905195 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:21.908500 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b04ae76622ac46f5a584a582e372bc05 in term 1.
I20250112 20:51:21.914270 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ae76622ac46f5a584a582e372bc05. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.914748 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:21.922103 684 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:21.924364 684 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:21.943981 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Refusing update from remote peer b04ae76622ac46f5a584a582e372bc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:21.945024 682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:21.952816 536 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Refusing update from remote peer b04ae76622ac46f5a584a582e372bc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:21.953711 682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:21.983484 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ae76622ac46f5a584a582e372bc05. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.983808 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:21.988324 685 mvcc.cc:204] Tried to move back new op lower bound from 7113584975614070784 to 7113584975441018880. Current Snapshot: MvccSnapshot[applied={T|T < 7113584975614070784}]
I20250112 20:51:21.989028 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.989257 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:21.997385 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:21.997721 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:21.999289 687 mvcc.cc:204] Tried to move back new op lower bound from 7113584975614070784 to 7113584975441018880. Current Snapshot: MvccSnapshot[applied={T|T < 7113584975614070784}]
I20250112 20:51:22.001147 684 catalog_manager.cc:1348] Generated new cluster ID: 55c7c6228b4d4e3e80a4f1e922af635d
I20250112 20:51:22.001245 684 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:22.014257 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ae76622ac46f5a584a582e372bc05. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:22.014552 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:22.015767 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:22.015978 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:22.029466 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:22.029783 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:22.058579 684 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:22.060288 684 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:22.080142 684 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Generated new TSK 0
I20250112 20:51:22.081219 684 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:22.113970 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250112 20:51:22.451892 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:22.452131 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:22.452250 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:22.459466 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:22.459672 694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:22.468467 694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:22.470036 694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:22.471695 694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:22.483595 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:22.484429 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:22.485567 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:22.486176 694 server_base.cc:1034] running on GCE node
I20250112 20:51:22.486809 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:22.487650 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:22.488883 694 hybrid_clock.cc:648] HybridClock initialized: now 1736715082488817 us; error 36 us; skew 500 ppm
I20250112 20:51:22.489138 694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:22.491555 694 webserver.cc:458] Webserver started at http://127.0.104.1:38009/ using document root <none> and password file <none>
I20250112 20:51:22.492211 694 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:22.492322 694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:22.492581 694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:22.495018 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data/instance:
uuid: "aefe1db8b19244d891b6bcd8971e9b17"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.495682 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/instance:
uuid: "aefe1db8b19244d891b6bcd8971e9b17"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.500322 694 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250112 20:51:22.503453 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:22.504616 694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:22.504745 694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal
uuid: "aefe1db8b19244d891b6bcd8971e9b17"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.504945 694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:22.528252 694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:22.529019 694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:22.529320 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:22.530247 694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 20:51:22.531693 694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:22.531759 694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:22.531872 694 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:22.531917 694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:22.564800 694 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35585
I20250112 20:51:22.564888 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35585 every 8 connection(s)
I20250112 20:51:22.566042 694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:22.574397 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 694
I20250112 20:51:22.574600 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:22.578666 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 20:51:22.582777 821 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:22.582885 822 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35199
I20250112 20:51:22.583143 822 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:22.583997 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:22.584132 823 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:22.584210 822 heartbeater.cc:510] Master 127.0.104.62:35199 requested a full tablet report, sending...
I20250112 20:51:22.584652 823 heartbeater.cc:510] Master 127.0.104.61:39271 requested a full tablet report, sending...
I20250112 20:51:22.583153 821 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:22.585357 821 heartbeater.cc:510] Master 127.0.104.60:45153 requested a full tablet report, sending...
I20250112 20:51:22.586321 455 ts_manager.cc:194] Registered new tserver with Master: aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:22.586356 526 ts_manager.cc:194] Registered new tserver with Master: aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:22.586756 597 ts_manager.cc:194] Registered new tserver with Master: aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:22.588498 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:47943
I20250112 20:51:22.589444 652 catalog_manager.cc:1260] Loaded cluster ID: 55c7c6228b4d4e3e80a4f1e922af635d
I20250112 20:51:22.589509 652 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: loading cluster ID for follower catalog manager: success
I20250112 20:51:22.592190 652 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: acquiring CA information for follower catalog manager: success
I20250112 20:51:22.593672 652 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 20:51:22.846273 680 catalog_manager.cc:1260] Loaded cluster ID: 55c7c6228b4d4e3e80a4f1e922af635d
I20250112 20:51:22.846396 680 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: loading cluster ID for follower catalog manager: success
I20250112 20:51:22.850230 680 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: acquiring CA information for follower catalog manager: success
I20250112 20:51:22.853165 680 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250112 20:51:22.921002 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:22.921288 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:22.921391 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:22.929852 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:22.930058 831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:22.939157 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:22.940673 831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:22.942292 831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:22.954373 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:22.955243 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:22.956158 839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:22.957116 831 server_base.cc:1034] running on GCE node
I20250112 20:51:22.957682 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:22.958600 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:22.959805 831 hybrid_clock.cc:648] HybridClock initialized: now 1736715082959746 us; error 37 us; skew 500 ppm
I20250112 20:51:22.960075 831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:22.962253 831 webserver.cc:458] Webserver started at http://127.0.104.2:33051/ using document root <none> and password file <none>
I20250112 20:51:22.962965 831 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:22.963081 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:22.963364 831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:22.965824 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data/instance:
uuid: "87ae45957c64469d96dc92d606b5b4b3"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.966964 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/instance:
uuid: "87ae45957c64469d96dc92d606b5b4b3"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.972044 831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20250112 20:51:22.975194 845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:22.976421 831 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:22.976555 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal
uuid: "87ae45957c64469d96dc92d606b5b4b3"
format_stamp: "Formatted at 2025-01-12 20:51:22 on dist-test-slave-4dz2"
I20250112 20:51:22.976723 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:23.010023 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:23.010856 831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:23.011124 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:23.012049 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 20:51:23.013710 831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:23.013780 831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:23.013895 831 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:23.013942 831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:23.048058 831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41673
I20250112 20:51:23.048096 957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41673 every 8 connection(s)
I20250112 20:51:23.051278 831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:23.053777 962 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:23.054024 962 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:23.055436 962 leader_election.cc:290] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271)
I20250112 20:51:23.056130 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:23.056524 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:23.057637 584 leader_election.cc:304] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5c09e899184de6b770dca873501062; no voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:23.058471 962 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:23.060214 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 831
I20250112 20:51:23.060451 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:23.068243 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 20:51:23.073760 959 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35199
I20250112 20:51:23.074124 959 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.075570 959 heartbeater.cc:510] Master 127.0.104.62:35199 requested a full tablet report, sending...
I20250112 20:51:23.076555 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:23.076686 960 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.077081 455 ts_manager.cc:194] Registered new tserver with Master: 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:23.077208 960 heartbeater.cc:510] Master 127.0.104.61:39271 requested a full tablet report, sending...
I20250112 20:51:23.078161 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:36661
I20250112 20:51:23.078922 958 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:23.078933 526 ts_manager.cc:194] Registered new tserver with Master: 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:23.079033 958 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.079493 958 heartbeater.cc:510] Master 127.0.104.60:45153 requested a full tablet report, sending...
I20250112 20:51:23.081462 597 ts_manager.cc:194] Registered new tserver with Master: 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:23.091188 822 heartbeater.cc:502] Master 127.0.104.62:35199 was elected leader, sending a full tablet report...
I20250112 20:51:23.393904 967 raft_consensus.cc:491] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:23.394106 967 raft_consensus.cc:513] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:23.395232 967 leader_election.cc:290] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153)
I20250112 20:51:23.395754 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b985fd656ce44131bc91ccfeb79587c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f5c09e899184de6b770dca873501062" is_pre_election: true
I20250112 20:51:23.395753 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b985fd656ce44131bc91ccfeb79587c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:23.396458 512 leader_election.cc:304] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b985fd656ce44131bc91ccfeb79587c5; no voters: 0f5c09e899184de6b770dca873501062, b04ae76622ac46f5a584a582e372bc05
I20250112 20:51:23.396860 967 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 20:51:23.409269 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:23.409500 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:23.409632 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:23.417228 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:23.417433 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:23.426250 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:23.427795 966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:23.429881 966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:23.441454 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:23.442441 973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:23.443295 975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:23.444080 966 server_base.cc:1034] running on GCE node
I20250112 20:51:23.444659 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:23.445526 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:23.446731 966 hybrid_clock.cc:648] HybridClock initialized: now 1736715083446674 us; error 30 us; skew 500 ppm
I20250112 20:51:23.447046 966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:23.449227 966 webserver.cc:458] Webserver started at http://127.0.104.3:43223/ using document root <none> and password file <none>
I20250112 20:51:23.449859 966 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:23.449985 966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:23.450273 966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:23.452716 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data/instance:
uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
format_stamp: "Formatted at 2025-01-12 20:51:23 on dist-test-slave-4dz2"
I20250112 20:51:23.453459 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/instance:
uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
format_stamp: "Formatted at 2025-01-12 20:51:23 on dist-test-slave-4dz2"
I20250112 20:51:23.458127 966 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250112 20:51:23.461647 981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:23.462877 966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:23.463033 966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal
uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
format_stamp: "Formatted at 2025-01-12 20:51:23 on dist-test-slave-4dz2"
I20250112 20:51:23.463196 966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:23.487639 966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:23.488427 966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:23.488927 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:23.489965 966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 20:51:23.491673 966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:23.491777 966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:23.491912 966 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:23.491969 966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:23.527076 966 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36593
I20250112 20:51:23.527109 1093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36593 every 8 connection(s)
I20250112 20:51:23.528370 966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:23.532258 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 966
I20250112 20:51:23.532461 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:23.544646 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35199
I20250112 20:51:23.544965 1095 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.545012 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:23.545135 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:23.545186 1094 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.545310 1096 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:23.546005 1096 heartbeater.cc:510] Master 127.0.104.61:39271 requested a full tablet report, sending...
I20250112 20:51:23.546031 1095 heartbeater.cc:510] Master 127.0.104.62:35199 requested a full tablet report, sending...
I20250112 20:51:23.546056 1094 heartbeater.cc:510] Master 127.0.104.60:45153 requested a full tablet report, sending...
I20250112 20:51:23.547302 455 ts_manager.cc:194] Registered new tserver with Master: 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:23.547358 526 ts_manager.cc:194] Registered new tserver with Master: 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:23.547556 597 ts_manager.cc:194] Registered new tserver with Master: 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:23.548220 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:59797
I20250112 20:51:23.557847 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:23.584192 959 heartbeater.cc:502] Master 127.0.104.62:35199 was elected leader, sending a full tablet report...
I20250112 20:51:23.586948 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b329a8ccba2e4f1ba96cd7fe7272c809"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:23.587093 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6bcf6c429a014aa8b66f92d5acf61afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:23.587116 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f90e27f877d14120a21eb86b7b1c783f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:23.588536 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_308cae5cf7534edcb43b50a13efb6c57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.589007 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b329a8ccba2e4f1ba96cd7fe7272c809 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:23.589191 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_308cae5cf7534edcb43b50a13efb6c57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:23.591361 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f90e27f877d14120a21eb86b7b1c783f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:23.592761 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bcf6c429a014aa8b66f92d5acf61afe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.593640 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_323461699c014be39bd3ccdb837aee61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.594192 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_323461699c014be39bd3ccdb837aee61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.594822 1112 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:23.594980 1112 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:23.596093 1112 leader_election.cc:290] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271)
I20250112 20:51:23.596166 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:23.597126 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:23.597709 584 leader_election.cc:304] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5c09e899184de6b770dca873501062; no voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:23.600551 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e95f075161dc4cb1bf8d635f2600e710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.601182 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e95f075161dc4cb1bf8d635f2600e710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.602489 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d950b5ed63bd4f6fb29a419c7c9d207a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.603209 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d950b5ed63bd4f6fb29a419c7c9d207a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.607766 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f2ea438ba8f643bb88063179ee0e4030"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.608748 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2ea438ba8f643bb88063179ee0e4030 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.609197 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a101728b27354355b705dc32ad44223b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.609740 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a101728b27354355b705dc32ad44223b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.613011 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7d7957d6055546aa982d5eb91d8d0538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.613620 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d7957d6055546aa982d5eb91d8d0538 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.623461 1112 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:23.660591 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_308cae5cf7534edcb43b50a13efb6c57" } new_table_name: "default.Table_516c544209364ebdad3fca4b714c6db1" modify_external_catalogs: true
I20250112 20:51:23.692600 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f2ea438ba8f643bb88063179ee0e4030" } modify_external_catalogs: true
I20250112 20:51:23.692903 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f90e27f877d14120a21eb86b7b1c783f" } new_table_name: "default.Table_d32f4745317e406797bf1b50c1e33102" modify_external_catalogs: true
I20250112 20:51:23.693290 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_323461699c014be39bd3ccdb837aee61" } new_table_name: "default.Table_ae6a03149fcb4b848a77bb2237a314ac" modify_external_catalogs: true
I20250112 20:51:23.693359 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e95f075161dc4cb1bf8d635f2600e710" } modify_external_catalogs: true
I20250112 20:51:23.693495 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e95f075161dc4cb1bf8d635f2600e710" } modify_external_catalogs: true
I20250112 20:51:23.694720 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d950b5ed63bd4f6fb29a419c7c9d207a" } modify_external_catalogs: true
I20250112 20:51:23.694793 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b329a8ccba2e4f1ba96cd7fe7272c809" } new_table_name: "default.Table_f3764d3dcba84aaa8597fa926a0dbe12" modify_external_catalogs: true
I20250112 20:51:23.694893 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d950b5ed63bd4f6fb29a419c7c9d207a" } modify_external_catalogs: true
I20250112 20:51:23.699368 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6bcf6c429a014aa8b66f92d5acf61afe" } new_table_name: "default.Table_8ff44a4d402d4e9097607704960eb04a" modify_external_catalogs: true
I20250112 20:51:23.704226 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a101728b27354355b705dc32ad44223b" } modify_external_catalogs: true
I20250112 20:51:23.704502 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a101728b27354355b705dc32ad44223b" } modify_external_catalogs: true
I20250112 20:51:23.705906 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7d7957d6055546aa982d5eb91d8d0538" } modify_external_catalogs: true
I20250112 20:51:23.706102 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7d7957d6055546aa982d5eb91d8d0538" } modify_external_catalogs: true
I20250112 20:51:23.692955 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f2ea438ba8f643bb88063179ee0e4030" } modify_external_catalogs: true
W20250112 20:51:23.725293 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab78028b7ec14798be944daebc738998 consensus state has no leader
I20250112 20:51:23.736083 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 85df469b28734cfd80d1076367827d7b
W20250112 20:51:23.736651 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 229da341da6d4b88853afcad32af3904 has no consensus state
W20250112 20:51:23.736716 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f125b1cc50643709f8666847c587266 has no consensus state
I20250112 20:51:23.737763 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 69fd7c4220054d75adaae9f794ae595d
W20250112 20:51:23.741487 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 has no consensus state
I20250112 20:51:23.742281 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8765ad1274194294b4f0562d70b8659d
I20250112 20:51:23.743142 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b171f6ddc413450994ed6543e592c173
W20250112 20:51:23.746526 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9898e024e23f4eb2ad38b7953dbd05d1 has no consensus state
I20250112 20:51:23.748239 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0f20e5f0eaad4586b61ecf10b104ecfb
W20250112 20:51:23.763370 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 229da341da6d4b88853afcad32af3904 consensus state has no leader
I20250112 20:51:23.762696 1028 tablet_service.cc:1467] Processing CreateTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (DEFAULT_TABLE table=default.Table_8ff44a4d402d4e9097607704960eb04a [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.762789 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 (DEFAULT_TABLE table=default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.764328 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ab78028b7ec14798be944daebc738998 (DEFAULT_TABLE table=default.Table_308cae5cf7534edcb43b50a13efb6c57 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.764478 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.764925 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 5f125b1cc50643709f8666847c587266 (DEFAULT_TABLE table=default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.762909 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 229da341da6d4b88853afcad32af3904 (DEFAULT_TABLE table=default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.766491 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f125b1cc50643709f8666847c587266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.766710 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229da341da6d4b88853afcad32af3904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.768008 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9898e024e23f4eb2ad38b7953dbd05d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.769328 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab78028b7ec14798be944daebc738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.769644 756 tablet_service.cc:1467] Processing CreateTablet for tablet ab78028b7ec14798be944daebc738998 (DEFAULT_TABLE table=default.Table_308cae5cf7534edcb43b50a13efb6c57 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.770501 753 tablet_service.cc:1467] Processing CreateTablet for tablet 229da341da6d4b88853afcad32af3904 (DEFAULT_TABLE table=default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.770752 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5f125b1cc50643709f8666847c587266 (DEFAULT_TABLE table=default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.769618 755 tablet_service.cc:1467] Processing CreateTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (DEFAULT_TABLE table=default.Table_8ff44a4d402d4e9097607704960eb04a [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.771095 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab78028b7ec14798be944daebc738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.771234 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 (DEFAULT_TABLE table=default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.771510 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f125b1cc50643709f8666847c587266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.771898 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9898e024e23f4eb2ad38b7953dbd05d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.772621 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229da341da6d4b88853afcad32af3904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.773533 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.790962 1163 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:23.795545 1163 tablet_bootstrap.cc:654] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.798046 1164 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:23.801941 1163 log.cc:826] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:23.804268 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu pid 492
I20250112 20:51:23.805588 1164 tablet_bootstrap.cc:654] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.805738 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:23.807106 1164 log.cc:826] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:23.807543 1163 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:23.807834 1163 ts_tablet_manager.cc:1397] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20250112 20:51:23.812513 1164 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:23.812764 1164 ts_tablet_manager.cc:1397] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20250112 20:51:23.818488 1163 raft_consensus.cc:357] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.818962 1163 raft_consensus.cc:383] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:23.819065 1163 raft_consensus.cc:738] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:23.818900 893 tablet_service.cc:1467] Processing CreateTablet for tablet ab78028b7ec14798be944daebc738998 (DEFAULT_TABLE table=default.Table_308cae5cf7534edcb43b50a13efb6c57 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.819900 1163 consensus_queue.cc:260] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.820466 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab78028b7ec14798be944daebc738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.822430 892 tablet_service.cc:1467] Processing CreateTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (DEFAULT_TABLE table=default.Table_8ff44a4d402d4e9097607704960eb04a [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.822465 1164 raft_consensus.cc:357] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.822867 1164 raft_consensus.cc:383] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:23.822916 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.822978 1164 raft_consensus.cc:738] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:23.823609 1164 consensus_queue.cc:260] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.829962 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 (DEFAULT_TABLE table=default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.830533 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9898e024e23f4eb2ad38b7953dbd05d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.834244 1163 ts_tablet_manager.cc:1428] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.026s user 0.009s sys 0.017s
I20250112 20:51:23.834568 1095 heartbeater.cc:502] Master 127.0.104.62:35199 was elected leader, sending a full tablet report...
I20250112 20:51:23.834683 1163 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:23.836555 1163 tablet_bootstrap.cc:654] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.839875 890 tablet_service.cc:1467] Processing CreateTablet for tablet 229da341da6d4b88853afcad32af3904 (DEFAULT_TABLE table=default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.840467 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229da341da6d4b88853afcad32af3904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.842301 1164 ts_tablet_manager.cc:1428] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.029s user 0.011s sys 0.017s
I20250112 20:51:23.842738 1164 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:23.844486 1164 tablet_bootstrap.cc:654] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.848924 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5f125b1cc50643709f8666847c587266 (DEFAULT_TABLE table=default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.852582 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]). Expected version 1 got 0
I20250112 20:51:23.854207 1163 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:23.854370 1163 ts_tablet_manager.cc:1397] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.015s
I20250112 20:51:23.855304 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]). Expected version 1 got 0
I20250112 20:51:23.855505 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 9898e024e23f4eb2ad38b7953dbd05d1 (table default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]). Expected version 1 got 0
I20250112 20:51:23.855449 1163 raft_consensus.cc:357] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.855681 1163 raft_consensus.cc:383] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:23.855688 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (table default.Table_8ff44a4d402d4e9097607704960eb04a [id=77c2c152845745ad8c8898a9547975d1]). Expected version 1 got 0
I20250112 20:51:23.855777 1163 raft_consensus.cc:738] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:23.855850 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]). Expected version 1 got 0
I20250112 20:51:23.856057 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet ab78028b7ec14798be944daebc738998 (table default.Table_516c544209364ebdad3fca4b714c6db1 [id=1d1adf4313fb4c69803023132dd314c2]). Expected version 1 got 0
I20250112 20:51:23.856046 1163 consensus_queue.cc:260] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.856664 1164 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:23.856886 1164 ts_tablet_manager.cc:1397] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20250112 20:51:23.857111 1163 ts_tablet_manager.cc:1428] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:23.857386 1163 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:23.857985 1164 raft_consensus.cc:357] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.858247 1164 raft_consensus.cc:383] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:23.858358 1164 raft_consensus.cc:738] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:23.858573 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f125b1cc50643709f8666847c587266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.858608 1164 consensus_queue.cc:260] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.859140 1163 tablet_bootstrap.cc:654] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.859382 1164 ts_tablet_manager.cc:1428] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:23.859678 1164 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:23.861266 1164 tablet_bootstrap.cc:654] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.863370 1164 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:23.863564 1164 ts_tablet_manager.cc:1397] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:51:23.864291 1164 raft_consensus.cc:357] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:23.865069 1164 raft_consensus.cc:383] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:23.865182 1164 raft_consensus.cc:738] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:23.865429 1164 consensus_queue.cc:260] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:23.867902 1164 ts_tablet_manager.cc:1428] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:23.869989 1164 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:23.871594 1164 tablet_bootstrap.cc:654] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:23.879427 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4911eabcac2c4a8b9d4b08525597082a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.880226 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4911eabcac2c4a8b9d4b08525597082a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.882588 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_87d81792a55c46fca891f92be097b350"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.883286 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87d81792a55c46fca891f92be097b350 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.891016 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ea40a9ca79cb4843a9153610e6b9c1f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.891752 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea40a9ca79cb4843a9153610e6b9c1f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.893880 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9b7759bef3d2447596dfe1da9f6e5e72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.894505 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b7759bef3d2447596dfe1da9f6e5e72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.898432 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9ff857d5fbad4e3dbe3e5afcd835918d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.899022 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff857d5fbad4e3dbe3e5afcd835918d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.903187 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_54895a2507b64553b62c2b8dbe48a747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.903756 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54895a2507b64553b62c2b8dbe48a747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.908088 1169 raft_consensus.cc:491] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:23.908363 1169 raft_consensus.cc:513] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.909188 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8524af8db8fe40ef83d2466d1e803fb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.909699 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8524af8db8fe40ef83d2466d1e803fb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.911020 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5595598433be4c89aa8aa789a148ab47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:23.911180 1169 leader_election.cc:290] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:23.911504 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5595598433be4c89aa8aa789a148ab47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.934429 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_780782d287f3495082023768c0ba6932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.935185 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_780782d287f3495082023768c0ba6932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.936753 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_59b3292413f941d1a834d8dc55a640bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:23.937328 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59b3292413f941d1a834d8dc55a640bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:23.955073 1168 raft_consensus.cc:491] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:23.955410 1168 raft_consensus.cc:513] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:23.958958 1168 leader_election.cc:290] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:23.985167 755 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (DEFAULT_TABLE table=default.Table_87d81792a55c46fca891f92be097b350 [id=fdf41f1792bb41e6bd764ee979d48a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.985802 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd7d40d429f4aa08fedf80d8acb42a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.989717 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (DEFAULT_TABLE table=default.Table_87d81792a55c46fca891f92be097b350 [id=fdf41f1792bb41e6bd764ee979d48a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.990135 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd7d40d429f4aa08fedf80d8acb42a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:23.992126 983 leader_election.cc:343] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9898e024e23f4eb2ad38b7953dbd05d1
I20250112 20:51:23.993326 888 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (DEFAULT_TABLE table=default.Table_87d81792a55c46fca891f92be097b350 [id=fdf41f1792bb41e6bd764ee979d48a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:23.993754 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d32f4745317e406797bf1b50c1e33102" } modify_external_catalogs: true
I20250112 20:51:23.993956 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d32f4745317e406797bf1b50c1e33102" } modify_external_catalogs: true
I20250112 20:51:23.994868 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8ff44a4d402d4e9097607704960eb04a" } new_table_name: "default.Table_0715c21f34a74a34ba9baeb1f71414d3" modify_external_catalogs: true
I20250112 20:51:23.995097 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_516c544209364ebdad3fca4b714c6db1" } new_table_name: "default.Table_6c0fcd020f404207804bc1fd94676779" modify_external_catalogs: true
I20250112 20:51:23.996793 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_87d81792a55c46fca891f92be097b350" } new_table_name: "default.Table_8f718d7e5075431fa4f4bbecf9c13e7d" modify_external_catalogs: true
I20250112 20:51:23.997160 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd7d40d429f4aa08fedf80d8acb42a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:23.998214 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f3764d3dcba84aaa8597fa926a0dbe12" } modify_external_catalogs: true
I20250112 20:51:23.998422 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f3764d3dcba84aaa8597fa926a0dbe12" } modify_external_catalogs: true
I20250112 20:51:23.999119 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae6a03149fcb4b848a77bb2237a314ac" } modify_external_catalogs: true
I20250112 20:51:23.999292 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae6a03149fcb4b848a77bb2237a314ac" } modify_external_catalogs: true
W20250112 20:51:24.030856 1097 tablet.cc:2367] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:24.063764 1169 raft_consensus.cc:491] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.064023 1169 raft_consensus.cc:513] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.064831 1169 leader_election.cc:290] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:24.066268 1178 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:24.071794 824 tablet.cc:2367] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:24.075695 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu with pid 492
I20250112 20:51:24.078919 1178 tablet_bootstrap.cc:654] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.082387 1178 log.cc:826] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:24.083912 1168 raft_consensus.cc:491] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.084136 1168 raft_consensus.cc:513] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.084959 1168 leader_election.cc:290] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:24.085901 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:24.092465 983 leader_election.cc:343] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:24.098404 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 consensus state has no leader
I20250112 20:51:24.099749 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ea40a9ca79cb4843a9153610e6b9c1f5" } new_table_name: "default.Table_b31913db5a3c4307b6ab12e6721af977" modify_external_catalogs: true
W20250112 20:51:24.101096 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dd7d40d429f4aa08fedf80d8acb42a7 consensus state has no leader
I20250112 20:51:24.101449 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b7759bef3d2447596dfe1da9f6e5e72" } modify_external_catalogs: true
I20250112 20:51:24.101661 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b7759bef3d2447596dfe1da9f6e5e72" } modify_external_catalogs: true
W20250112 20:51:24.102352 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab78028b7ec14798be944daebc738998 consensus state has no leader
I20250112 20:51:24.102713 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8524af8db8fe40ef83d2466d1e803fb4" } new_table_name: "default.Table_a1d08a9783214c518f1eda3d907286a1" modify_external_catalogs: true
I20250112 20:51:24.104912 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ff857d5fbad4e3dbe3e5afcd835918d" } modify_external_catalogs: true
I20250112 20:51:24.105113 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ff857d5fbad4e3dbe3e5afcd835918d" } modify_external_catalogs: true
I20250112 20:51:24.106300 446 master_service.cc:946] ReplaceTablet: received request to replace tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.114424 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]). Expected version 1 got 0
I20250112 20:51:24.116097 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]). Expected version 1 got 0
I20250112 20:51:24.126904 754 tablet_service.cc:1467] Processing CreateTablet for tablet 87756828be274286a3bb674040727828 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.127457 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87756828be274286a3bb674040727828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.128113 887 tablet_service.cc:1467] Processing CreateTablet for tablet e3a895feb0504d39ba8f93b25e2e032d (DEFAULT_TABLE table=default.Table_8524af8db8fe40ef83d2466d1e803fb4 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.128526 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79679bd64041e69e8c15369ec4acfb (DEFAULT_TABLE table=default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.128693 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a895feb0504d39ba8f93b25e2e032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.129040 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f79679bd64041e69e8c15369ec4acfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.129148 1026 tablet_service.cc:1467] Processing CreateTablet for tablet adb0973b4ae8457b888d614682f52b62 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.129508 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb0973b4ae8457b888d614682f52b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.130844 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbcad11f1934759804e51554cd05e3d (DEFAULT_TABLE table=default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.131526 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbcad11f1934759804e51554cd05e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.133090 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 87756828be274286a3bb674040727828 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.133440 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87756828be274286a3bb674040727828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.136631 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e482a2543cdf4020a03f131c980e7928 (DEFAULT_TABLE table=default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.137027 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbcad11f1934759804e51554cd05e3d (DEFAULT_TABLE table=default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.137601 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79679bd64041e69e8c15369ec4acfb (DEFAULT_TABLE table=default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.137662 893 tablet_service.cc:1467] Processing CreateTablet for tablet e482a2543cdf4020a03f131c980e7928 (DEFAULT_TABLE table=default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.138068 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e482a2543cdf4020a03f131c980e7928. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.138337 1027 tablet_service.cc:1467] Processing CreateTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 (DEFAULT_TABLE table=default.Table_ea40a9ca79cb4843a9153610e6b9c1f5 [id=a578996c09484fb1a6b08eb55eff74d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.138949 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e482a2543cdf4020a03f131c980e7928. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.141135 888 tablet_service.cc:1467] Processing CreateTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 (DEFAULT_TABLE table=default.Table_ea40a9ca79cb4843a9153610e6b9c1f5 [id=a578996c09484fb1a6b08eb55eff74d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.141551 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9fcf5bc7584cdcb17b19fc69802b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.143854 756 tablet_service.cc:1467] Processing CreateTablet for tablet e482a2543cdf4020a03f131c980e7928 (DEFAULT_TABLE table=default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.144255 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e482a2543cdf4020a03f131c980e7928. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.144457 752 tablet_service.cc:1467] Processing CreateTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 (DEFAULT_TABLE table=default.Table_ea40a9ca79cb4843a9153610e6b9c1f5 [id=a578996c09484fb1a6b08eb55eff74d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.145100 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbcad11f1934759804e51554cd05e3d (DEFAULT_TABLE table=default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.145156 751 tablet_service.cc:1467] Processing CreateTablet for tablet 6f79679bd64041e69e8c15369ec4acfb (DEFAULT_TABLE table=default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.145491 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbcad11f1934759804e51554cd05e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.146435 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbcad11f1934759804e51554cd05e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.147254 892 tablet_service.cc:1467] Processing CreateTablet for tablet 87756828be274286a3bb674040727828 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.147549 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87756828be274286a3bb674040727828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.149287 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f79679bd64041e69e8c15369ec4acfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.151304 750 tablet_service.cc:1467] Processing CreateTablet for tablet e3a895feb0504d39ba8f93b25e2e032d (DEFAULT_TABLE table=default.Table_8524af8db8fe40ef83d2466d1e803fb4 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.151770 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a895feb0504d39ba8f93b25e2e032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.153218 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f79679bd64041e69e8c15369ec4acfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.153261 889 tablet_service.cc:1467] Processing CreateTablet for tablet adb0973b4ae8457b888d614682f52b62 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.153594 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb0973b4ae8457b888d614682f52b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.156350 755 tablet_service.cc:1467] Processing CreateTablet for tablet adb0973b4ae8457b888d614682f52b62 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.156656 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9fcf5bc7584cdcb17b19fc69802b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.156821 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adb0973b4ae8457b888d614682f52b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.159866 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9fcf5bc7584cdcb17b19fc69802b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.165082 1178 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.165374 1178 ts_tablet_manager.cc:1397] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.099s user 0.010s sys 0.019s
W20250112 20:51:24.170128 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3a895feb0504d39ba8f93b25e2e032d consensus state has no leader
I20250112 20:51:24.170423 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 229da341da6d4b88853afcad32af3904 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.171631 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:24.172040 1049 raft_consensus.cc:2463] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:24.172443 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229da341da6d4b88853afcad32af3904" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:24.177084 1178 raft_consensus.cc:357] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.177526 1178 raft_consensus.cc:383] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.177686 1178 raft_consensus.cc:738] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.178529 1178 consensus_queue.cc:260] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:24.184090 849 connection.cc:537] client connection to 127.0.104.61:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:24.184504 985 connection.cc:537] client connection to 127.0.104.61:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:24.184649 960 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:39271 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:24.185142 1096 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:39271 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:24.188308 823 heartbeater.cc:643] Failed to heartbeat to 127.0.104.61:39271 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.61:39271 (error 108)
I20250112 20:51:24.190580 1178 ts_tablet_manager.cc:1428] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.025s user 0.008s sys 0.007s
W20250112 20:51:24.195358 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df9fcf5bc7584cdcb17b19fc69802b84 consensus state has no leader
I20250112 20:51:24.195660 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 229da341da6d4b88853afcad32af3904 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.197275 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5f125b1cc50643709f8666847c587266
I20250112 20:51:24.197489 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 229da341da6d4b88853afcad32af3904
I20250112 20:51:24.197777 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 229da341da6d4b88853afcad32af3904 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.198184 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 229da341da6d4b88853afcad32af3904 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.198351 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_954436533cfc462dbffed1710e7920ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:24.198705 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0a1da2b7cee84bd49753e13ddab19768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.198894 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_954436533cfc462dbffed1710e7920ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:24.199156 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a1da2b7cee84bd49753e13ddab19768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.200533 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9ea30f59545d4a81929c403c64374844"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.201151 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea30f59545d4a81929c403c64374844 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.201828 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet ab78028b7ec14798be944daebc738998 (table default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]). Expected version 2 got 0
I20250112 20:51:24.202759 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_073e7025c8464101ace706c4fe1e5c2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.203352 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073e7025c8464101ace706c4fe1e5c2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.203759 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e3a895feb0504d39ba8f93b25e2e032d (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.204213 1178 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.205391 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a895feb0504d39ba8f93b25e2e032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.205941 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f125b1cc50643709f8666847c587266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.207391 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f125b1cc50643709f8666847c587266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.208009 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 229da341da6d4b88853afcad32af3904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.208122 1184 tablet_replica.cc:331] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.208680 1184 raft_consensus.cc:2238] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.209904 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0c9b5c2846724b71a49b34852f5bf4aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.210415 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c9b5c2846724b71a49b34852f5bf4aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.209123 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 229da341da6d4b88853afcad32af3904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.213582 1184 raft_consensus.cc:2267] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.210351 1178 tablet_bootstrap.cc:654] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:24.217660 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5f125b1cc50643709f8666847c587266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f125b1cc50643709f8666847c587266 already in progress: creating tablet
I20250112 20:51:24.217651 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f125b1cc50643709f8666847c587266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.219589 1184 ts_tablet_manager.cc:1905] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.220731 710 leader_election.cc:304] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:24.221697 1185 tablet_replica.cc:331] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.221992 1169 raft_consensus.cc:2798] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:24.222146 1169 raft_consensus.cc:491] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:24.222210 1185 raft_consensus.cc:2238] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.222283 1169 raft_consensus.cc:3054] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.222605 1185 raft_consensus.cc:2267] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.223999 1178 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.224138 1178 ts_tablet_manager.cc:1397] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250112 20:51:24.225111 1178 raft_consensus.cc:357] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.225312 1178 raft_consensus.cc:383] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.225394 1178 raft_consensus.cc:738] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.225832 1169 raft_consensus.cc:513] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.226598 1169 leader_election.cc:290] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:24.226706 1185 ts_tablet_manager.cc:1905] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:24.227253 710 leader_election.cc:343] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.221719 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 229da341da6d4b88853afcad32af3904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.228269 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:24.229828 1163 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.229995 1163 ts_tablet_manager.cc:1397] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.373s user 0.097s sys 0.193s
W20250112 20:51:24.230682 1183 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 20:51:24.195669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:39271 (local address 127.0.104.2:41867)
0112 20:51:24.229244 (+ 33575us) negotiation.cc:107] Waiting for socket to connect
0112 20:51:24.229456 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33432,"thread_start_us":34258,"threads_started":1}
I20250112 20:51:24.230767 1163 raft_consensus.cc:357] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.230966 1163 raft_consensus.cc:383] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.231046 1163 raft_consensus.cc:738] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:24.231743 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 229da341da6d4b88853afcad32af3904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 229da341da6d4b88853afcad32af3904 already in progress: creating tablet
W20250112 20:51:24.231819 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39271, user_credentials={real_user=slave}} blocked reactor thread for 43495.1us
W20250112 20:51:24.233026 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271): Couldn't send request to peer b985fd656ce44131bc91ccfeb79587c5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:24.233309 1163 consensus_queue.cc:260] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.236258 1163 ts_tablet_manager.cc:1428] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250112 20:51:24.236543 1163 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.238643 1184 ts_tablet_manager.cc:1918] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.238734 1163 tablet_bootstrap.cc:654] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.238786 1184 log.cc:1198] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5f125b1cc50643709f8666847c587266
I20250112 20:51:24.239240 1184 ts_tablet_manager.cc:1939] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.239300 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9898e024e23f4eb2ad38b7953dbd05d1" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:24.240182 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]) successfully deleted
I20250112 20:51:24.240607 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:24.231199 1178 consensus_queue.cc:260] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.241024 776 raft_consensus.cc:2388] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
W20250112 20:51:24.241696 983 leader_election.cc:343] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.241860 983 leader_election.cc:304] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:24.242693 1168 raft_consensus.cc:2743] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:24.243521 983 leader_election.cc:304] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:24.243777 983 raft_consensus.cc:2681] T 5f125b1cc50643709f8666847c587266 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:24.244028 1178 ts_tablet_manager.cc:1428] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
W20250112 20:51:24.245141 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 229da341da6d4b88853afcad32af3904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 229da341da6d4b88853afcad32af3904 already in progress: creating tablet
I20250112 20:51:24.245319 1178 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:24.245945 710 leader_election.cc:343] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:24.247550 1178 tablet_bootstrap.cc:654] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.249753 1164 tablet_bootstrap.cc:492] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.249907 1164 ts_tablet_manager.cc:1397] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.380s user 0.109s sys 0.126s
I20250112 20:51:24.250726 1164 raft_consensus.cc:357] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.250918 1164 raft_consensus.cc:383] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.250998 1164 raft_consensus.cc:738] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.251231 1185 ts_tablet_manager.cc:1918] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.251210 1164 consensus_queue.cc:260] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.251369 1185 log.cc:1198] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/229da341da6d4b88853afcad32af3904
I20250112 20:51:24.251850 1185 ts_tablet_manager.cc:1939] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.252861 1164 ts_tablet_manager.cc:1428] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:24.253175 1164 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.253585 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]) successfully deleted
I20250112 20:51:24.254752 1178 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.254899 1178 ts_tablet_manager.cc:1397] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20250112 20:51:24.255887 1178 raft_consensus.cc:357] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.256100 1178 raft_consensus.cc:383] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.256171 1178 raft_consensus.cc:738] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.256359 1163 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.256494 1163 ts_tablet_manager.cc:1397] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20250112 20:51:24.256402 1178 consensus_queue.cc:260] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.256919 1178 ts_tablet_manager.cc:1428] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 20:51:24.257196 1178 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.257268 1163 raft_consensus.cc:357] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.257457 1163 raft_consensus.cc:383] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.257541 1163 raft_consensus.cc:738] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.257758 1163 consensus_queue.cc:260] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.258216 1163 ts_tablet_manager.cc:1428] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:24.258440 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9898e024e23f4eb2ad38b7953dbd05d1
I20250112 20:51:24.258494 1163 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.258772 1178 tablet_bootstrap.cc:654] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.259687 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6f79679bd64041e69e8c15369ec4acfb
I20250112 20:51:24.260061 1163 tablet_bootstrap.cc:654] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.261713 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_886e8c220c4c495c9de8c57e28750a64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.262372 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_886e8c220c4c495c9de8c57e28750a64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.263814 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:24.265273 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.265750 446 catalog_manager.cc:6596] ReplaceTablet: tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 deleted and replaced by tablet b07cef7306074ba3b7f8606bdfa6204a
I20250112 20:51:24.266145 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_edf0cd8097e1425a86a5ef68749d2a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.266718 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edf0cd8097e1425a86a5ef68749d2a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.271795 1163 tablet_bootstrap.cc:492] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.271948 1163 ts_tablet_manager.cc:1397] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20250112 20:51:24.272697 1163 raft_consensus.cc:357] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.273209 1185 tablet_replica.cc:331] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.273202 1163 raft_consensus.cc:383] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.273281 1163 raft_consensus.cc:738] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.273523 1163 consensus_queue.cc:260] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.273835 1169 raft_consensus.cc:491] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.274557 1185 raft_consensus.cc:2238] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.274737 1185 raft_consensus.cc:2267] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.275825 1169 raft_consensus.cc:513] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.276391 1163 ts_tablet_manager.cc:1428] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:24.276383 1169 leader_election.cc:290] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:24.276974 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab78028b7ec14798be944daebc738998" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:24.277148 1048 raft_consensus.cc:2463] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:24.276994 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229da341da6d4b88853afcad32af3904" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:24.277226 1163 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.277483 912 raft_consensus.cc:2463] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:24.277881 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:24.276675 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f125b1cc50643709f8666847c587266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:24.278940 1163 tablet_bootstrap.cc:654] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.279028 710 leader_election.cc:304] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:24.279137 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab78028b7ec14798be944daebc738998" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:24.280473 710 raft_consensus.cc:2681] T 229da341da6d4b88853afcad32af3904 P aefe1db8b19244d891b6bcd8971e9b17: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:24.281147 1185 ts_tablet_manager.cc:1905] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.282734 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.282996 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.283238 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.283438 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 with delete_type TABLET_DATA_DELETED (replaced by tablet b07cef7306074ba3b7f8606bdfa6204a) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.284389 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.284691 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.284897 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.285085 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 with delete_type TABLET_DATA_DELETED (replaced by tablet b07cef7306074ba3b7f8606bdfa6204a) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.285367 1186 tablet_replica.cc:331] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.285918 1186 raft_consensus.cc:2238] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.286151 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9898e024e23f4eb2ad38b7953dbd05d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.286350 1186 raft_consensus.cc:2267] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.286443 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.286675 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.286885 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 with delete_type TABLET_DATA_DELETED (replaced by tablet b07cef7306074ba3b7f8606bdfa6204a) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.286448 1184 tablet_replica.cc:331] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.287379 1184 raft_consensus.cc:2238] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.287560 1184 raft_consensus.cc:2267] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.288327 1184 ts_tablet_manager.cc:1905] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.264593 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5401cf6f9421453fbe9e59937a098182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.293694 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5401cf6f9421453fbe9e59937a098182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:24.294287 710 leader_election.cc:343] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.295993 1186 ts_tablet_manager.cc:1905] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.302208 710 leader_election.cc:304] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:24.302526 710 raft_consensus.cc:2681] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 20:51:24.304597 710 leader_election.cc:343] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.305115 1168 raft_consensus.cc:491] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.305294 1168 raft_consensus.cc:513] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:24.305410 961 tablet.cc:2367] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:24.305884 1168 leader_election.cc:290] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:24.305042 710 leader_election.cc:343] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.307443 1164 tablet_bootstrap.cc:654] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.308017 710 leader_election.cc:304] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:24.310262 1169 raft_consensus.cc:2798] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:24.310397 1169 raft_consensus.cc:491] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:24.310600 1169 raft_consensus.cc:3054] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.310882 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229da341da6d4b88853afcad32af3904" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:24.311120 913 raft_consensus.cc:2463] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:24.311713 983 leader_election.cc:304] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:24.312057 1168 raft_consensus.cc:2798] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:24.312172 1168 raft_consensus.cc:491] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:24.312284 1168 raft_consensus.cc:3054] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.314100 1169 raft_consensus.cc:513] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.314785 1168 raft_consensus.cc:513] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.315330 1168 leader_election.cc:290] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:24.317518 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab78028b7ec14798be944daebc738998" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:24.317691 1048 raft_consensus.cc:3054] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.317870 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab78028b7ec14798be944daebc738998" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:24.319046 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229da341da6d4b88853afcad32af3904" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:24.319294 913 raft_consensus.cc:3054] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.320719 1048 raft_consensus.cc:2463] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:24.321027 983 leader_election.cc:343] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 229da341da6d4b88853afcad32af3904
W20250112 20:51:24.321733 983 leader_election.cc:343] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 229da341da6d4b88853afcad32af3904
W20250112 20:51:24.322073 710 leader_election.cc:343] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.323184 1184 ts_tablet_manager.cc:1918] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.323315 1184 log.cc:1198] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9898e024e23f4eb2ad38b7953dbd05d1
I20250112 20:51:24.324921 1168 raft_consensus.cc:491] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.325871 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_780782d287f3495082023768c0ba6932" } new_table_name: "default.Table_4c35fe9b46fd41b78ea53fd616abc99b" modify_external_catalogs: true
I20250112 20:51:24.325109 1168 raft_consensus.cc:513] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.326905 1168 leader_election.cc:290] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:24.327227 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6a23debfb16c4480b703dc234f76eae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:24.324721 913 raft_consensus.cc:2463] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:24.327728 1169 leader_election.cc:290] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:24.327761 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a23debfb16c4480b703dc234f76eae5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.328372 710 leader_election.cc:304] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:24.329085 1169 raft_consensus.cc:2798] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:24.330169 983 leader_election.cc:304] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:24.330255 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f8ae4b86db714d6bba40875699ee81a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:24.330686 1168 raft_consensus.cc:2798] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:24.330616 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b31913db5a3c4307b6ab12e6721af977" } new_table_name: "default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1" modify_external_catalogs: true
W20250112 20:51:24.330739 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8ae4b86db714d6bba40875699ee81a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.331506 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad27af9a5df74c5a86bc7b0b05d9f8f3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:24.331794 913 raft_consensus.cc:2463] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:24.332943 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_073e7025c8464101ace706c4fe1e5c2f" } new_table_name: "default.Table_e7baae11d0fa447dbca4bf5211e3939c" modify_external_catalogs: true
I20250112 20:51:24.335268 1185 ts_tablet_manager.cc:1918] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.335439 1185 log.cc:1198] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5f125b1cc50643709f8666847c587266
I20250112 20:51:24.335844 1185 ts_tablet_manager.cc:1939] T 5f125b1cc50643709f8666847c587266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.338160 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 (DEFAULT_TABLE table=default.Table_59b3292413f941d1a834d8dc55a640bc [id=8ffaa9b9ac214c9f9d80dcc8865287a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.338630 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e814b7a9b4cb4336b62b7cd7e35461b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.341118 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (DEFAULT_TABLE table=default.Table_780782d287f3495082023768c0ba6932 [id=19853915fa604539b91aab6a4dc926e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.341446 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e040383f6e2a4ccf9614fd2d119a3cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.341861 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad27af9a5df74c5a86bc7b0b05d9f8f3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:24.342484 983 leader_election.cc:343] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:24.343135 983 leader_election.cc:304] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:24.348385 888 tablet_service.cc:1467] Processing CreateTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 (DEFAULT_TABLE table=default.Table_59b3292413f941d1a834d8dc55a640bc [id=8ffaa9b9ac214c9f9d80dcc8865287a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.348965 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e814b7a9b4cb4336b62b7cd7e35461b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.353099 893 tablet_service.cc:1467] Processing CreateTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (DEFAULT_TABLE table=default.Table_780782d287f3495082023768c0ba6932 [id=19853915fa604539b91aab6a4dc926e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.353531 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e040383f6e2a4ccf9614fd2d119a3cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.357229 755 tablet_service.cc:1467] Processing CreateTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 (DEFAULT_TABLE table=default.Table_59b3292413f941d1a834d8dc55a640bc [id=8ffaa9b9ac214c9f9d80dcc8865287a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.357785 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e814b7a9b4cb4336b62b7cd7e35461b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:24.358081 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e040383f6e2a4ccf9614fd2d119a3cb4 consensus state has no leader
I20250112 20:51:24.360983 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 229da341da6d4b88853afcad32af3904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:24.361207 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24abf89a9b2542d7bac2f7499698df3a has no consensus state
I20250112 20:51:24.361522 753 tablet_service.cc:1467] Processing CreateTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (DEFAULT_TABLE table=default.Table_780782d287f3495082023768c0ba6932 [id=19853915fa604539b91aab6a4dc926e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.362020 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e040383f6e2a4ccf9614fd2d119a3cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.377367 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0a1da2b7cee84bd49753e13ddab19768" } new_table_name: "default.Table_d38bb6feb1d44f2fbff02b9530af3908" modify_external_catalogs: true
W20250112 20:51:24.378222 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df9fcf5bc7584cdcb17b19fc69802b84 consensus state has no leader
W20250112 20:51:24.384022 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7df227e3389841259dc1420ec800e1c1 has no consensus state
I20250112 20:51:24.389770 1178 tablet_bootstrap.cc:492] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.390064 1178 ts_tablet_manager.cc:1397] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.133s user 0.046s sys 0.081s
I20250112 20:51:24.390411 1184 ts_tablet_manager.cc:1939] T 9898e024e23f4eb2ad38b7953dbd05d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.391338 1178 raft_consensus.cc:357] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.391973 1184 tablet_replica.cc:331] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.392210 1184 raft_consensus.cc:2238] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.392400 1184 raft_consensus.cc:2267] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.392829 1168 raft_consensus.cc:695] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:24.393311 1168 consensus_queue.cc:237] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:24.394385 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 51252.5us
I20250112 20:51:24.395889 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9898e024e23f4eb2ad38b7953dbd05d1 (table default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]) successfully deleted
W20250112 20:51:24.396179 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6f79679bd64041e69e8c15369ec4acfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f79679bd64041e69e8c15369ec4acfb already in progress: creating tablet
W20250112 20:51:24.396406 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9fbcad11f1934759804e51554cd05e3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fbcad11f1934759804e51554cd05e3d already in progress: creating tablet
I20250112 20:51:24.401278 1169 raft_consensus.cc:695] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:24.401661 1178 raft_consensus.cc:383] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.401873 1178 raft_consensus.cc:738] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.402233 1169 consensus_queue.cc:237] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.402206 1178 consensus_queue.cc:260] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.403061 1178 ts_tablet_manager.cc:1428] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 20:51:24.405125 1178 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.406836 1178 tablet_bootstrap.cc:654] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.414815 1184 ts_tablet_manager.cc:1905] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.423161 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7df227e3389841259dc1420ec800e1c1 (DEFAULT_TABLE table=default.Table_d38bb6feb1d44f2fbff02b9530af3908 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.423565 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df227e3389841259dc1420ec800e1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.423785 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 (DEFAULT_TABLE table=default.Table_edf0cd8097e1425a86a5ef68749d2a6c [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.424165 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.428126 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1c31c29073aa46ffa847905be1595a78 (DEFAULT_TABLE table=default.Table_0c9b5c2846724b71a49b34852f5bf4aa [id=c6a4acd610364581857563f69a97d6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.428540 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c31c29073aa46ffa847905be1595a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.429064 752 tablet_service.cc:1467] Processing CreateTablet for tablet b07cef7306074ba3b7f8606bdfa6204a (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.429385 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07cef7306074ba3b7f8606bdfa6204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.433009 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7df227e3389841259dc1420ec800e1c1 (DEFAULT_TABLE table=default.Table_d38bb6feb1d44f2fbff02b9530af3908 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.433017 754 tablet_service.cc:1467] Processing CreateTablet for tablet c7a57ea399194a45bc8990a029550440 (DEFAULT_TABLE table=default.Table_9ea30f59545d4a81929c403c64374844 [id=8133cc129a404fb8aca53d1fdb1c7fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.433319 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a57ea399194a45bc8990a029550440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.433346 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df227e3389841259dc1420ec800e1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.437011 750 tablet_service.cc:1467] Processing CreateTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 (DEFAULT_TABLE table=default.Table_886e8c220c4c495c9de8c57e28750a64 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.437278 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e53e67076c4a5a89f1e93fc168a997. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.438361 887 tablet_service.cc:1467] Processing CreateTablet for tablet c7a57ea399194a45bc8990a029550440 (DEFAULT_TABLE table=default.Table_9ea30f59545d4a81929c403c64374844 [id=8133cc129a404fb8aca53d1fdb1c7fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.438479 889 tablet_service.cc:1467] Processing CreateTablet for tablet b07cef7306074ba3b7f8606bdfa6204a (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.438747 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f125b1cc50643709f8666847c587266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.438896 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a57ea399194a45bc8990a029550440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.439229 885 tablet_service.cc:1467] Processing CreateTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 (DEFAULT_TABLE table=default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.439751 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3bd4eb92114bdb98b923fda9192cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.441032 756 tablet_service.cc:1467] Processing CreateTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 (DEFAULT_TABLE table=default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.441321 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3bd4eb92114bdb98b923fda9192cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.445045 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07cef7306074ba3b7f8606bdfa6204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.446451 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1c31c29073aa46ffa847905be1595a78 (DEFAULT_TABLE table=default.Table_0c9b5c2846724b71a49b34852f5bf4aa [id=c6a4acd610364581857563f69a97d6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.446789 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c31c29073aa46ffa847905be1595a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.449045 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7df227e3389841259dc1420ec800e1c1 (DEFAULT_TABLE table=default.Table_d38bb6feb1d44f2fbff02b9530af3908 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.449142 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 (DEFAULT_TABLE table=default.Table_edf0cd8097e1425a86a5ef68749d2a6c [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.449362 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df227e3389841259dc1420ec800e1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.449553 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.450001 886 tablet_service.cc:1467] Processing CreateTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 (DEFAULT_TABLE table=default.Table_886e8c220c4c495c9de8c57e28750a64 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.450513 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e53e67076c4a5a89f1e93fc168a997. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.451753 884 tablet_service.cc:1467] Processing CreateTablet for tablet 24abf89a9b2542d7bac2f7499698df3a (DEFAULT_TABLE table=default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.452190 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24abf89a9b2542d7bac2f7499698df3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.452968 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c7a57ea399194a45bc8990a029550440 (DEFAULT_TABLE table=default.Table_9ea30f59545d4a81929c403c64374844 [id=8133cc129a404fb8aca53d1fdb1c7fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.453374 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a57ea399194a45bc8990a029550440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.453373 749 tablet_service.cc:1467] Processing CreateTablet for tablet 24abf89a9b2542d7bac2f7499698df3a (DEFAULT_TABLE table=default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.453823 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24abf89a9b2542d7bac2f7499698df3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.457716 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b07cef7306074ba3b7f8606bdfa6204a (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.458030 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07cef7306074ba3b7f8606bdfa6204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.461989 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 (DEFAULT_TABLE table=default.Table_886e8c220c4c495c9de8c57e28750a64 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.462414 1022 tablet_service.cc:1467] Processing CreateTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 (DEFAULT_TABLE table=default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.463055 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e53e67076c4a5a89f1e93fc168a997. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.466606 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 24abf89a9b2542d7bac2f7499698df3a (DEFAULT_TABLE table=default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.467547 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3bd4eb92114bdb98b923fda9192cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.471187 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24abf89a9b2542d7bac2f7499698df3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.471953 755 tablet_service.cc:1467] Processing CreateTablet for tablet 1c31c29073aa46ffa847905be1595a78 (DEFAULT_TABLE table=default.Table_0c9b5c2846724b71a49b34852f5bf4aa [id=c6a4acd610364581857563f69a97d6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.472335 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c31c29073aa46ffa847905be1595a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.477100 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8f718d7e5075431fa4f4bbecf9c13e7d" } new_table_name: "default.Table_c9327af43e4f40429a836542ade34ab7" modify_external_catalogs: true
W20250112 20:51:24.485129 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dd7d40d429f4aa08fedf80d8acb42a7 consensus state has no leader
I20250112 20:51:24.493173 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e3a895feb0504d39ba8f93b25e2e032d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.497946 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f79679bd64041e69e8c15369ec4acfb from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.498782 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5595598433be4c89aa8aa789a148ab47" } modify_external_catalogs: true
I20250112 20:51:24.498970 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5595598433be4c89aa8aa789a148ab47" } modify_external_catalogs: true
I20250112 20:51:24.499470 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 (DEFAULT_TABLE table=default.Table_edf0cd8097e1425a86a5ef68749d2a6c [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.499856 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.501480 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ab78028b7ec14798be944daebc738998 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.503959 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e3a895feb0504d39ba8f93b25e2e032d
I20250112 20:51:24.504441 449 catalog_manager.cc:6596] ReplaceTablet: tablet e3a895feb0504d39ba8f93b25e2e032d deleted and replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:24.507211 1185 tablet_replica.cc:331] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.507392 1185 raft_consensus.cc:2238] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.507577 1185 raft_consensus.cc:2267] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.508530 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e7baae11d0fa447dbca4bf5211e3939c" } modify_external_catalogs: true
I20250112 20:51:24.508713 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e7baae11d0fa447dbca4bf5211e3939c" } modify_external_catalogs: true
I20250112 20:51:24.511396 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e482a2543cdf4020a03f131c980e7928
I20250112 20:51:24.512184 1186 ts_tablet_manager.cc:1918] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.512367 1186 log.cc:1198] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9898e024e23f4eb2ad38b7953dbd05d1
I20250112 20:51:24.512852 1186 ts_tablet_manager.cc:1939] T 9898e024e23f4eb2ad38b7953dbd05d1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.516135 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ab78028b7ec14798be944daebc738998
I20250112 20:51:24.516610 450 catalog_manager.cc:6596] ReplaceTablet: tablet ab78028b7ec14798be944daebc738998 deleted and replaced by tablet eb1dffbe43ba4eeab7ee12d901b62ffb
I20250112 20:51:24.517175 1164 tablet_bootstrap.cc:492] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.517395 1164 ts_tablet_manager.cc:1397] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.014s
I20250112 20:51:24.518252 1164 raft_consensus.cc:357] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.518502 1164 raft_consensus.cc:383] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.518584 1164 raft_consensus.cc:738] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.519248 1164 consensus_queue.cc:260] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.519660 1186 tablet_replica.cc:331] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.519869 1186 raft_consensus.cc:2238] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.520030 1186 raft_consensus.cc:2267] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.521286 1186 ts_tablet_manager.cc:1905] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.522431 1164 ts_tablet_manager.cc:1428] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:24.523394 1164 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.533732 1186 ts_tablet_manager.cc:1918] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.533893 1186 log.cc:1198] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.541128 1186 ts_tablet_manager.cc:1939] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.544551 1186 tablet_replica.cc:331] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.544735 1186 raft_consensus.cc:2238] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.544924 1186 raft_consensus.cc:2267] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.545862 1164 tablet_bootstrap.cc:654] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.546939 1186 ts_tablet_manager.cc:1905] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.547888 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 24abf89a9b2542d7bac2f7499698df3a
I20250112 20:51:24.550108 1164 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.550240 1164 ts_tablet_manager.cc:1397] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250112 20:51:24.550976 1164 raft_consensus.cc:357] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.551172 1164 raft_consensus.cc:383] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.551247 1164 raft_consensus.cc:738] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.552981 455 master_service.cc:946] ReplaceTablet: received request to replace tablet adb0973b4ae8457b888d614682f52b62 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.553072 1164 consensus_queue.cc:260] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.556960 1164 ts_tablet_manager.cc:1428] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:24.557850 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_54895a2507b64553b62c2b8dbe48a747" } modify_external_catalogs: true
I20250112 20:51:24.558068 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_54895a2507b64553b62c2b8dbe48a747" } modify_external_catalogs: true
I20250112 20:51:24.558696 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6a23debfb16c4480b703dc234f76eae5" } modify_external_catalogs: true
I20250112 20:51:24.558873 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6a23debfb16c4480b703dc234f76eae5" } modify_external_catalogs: true
I20250112 20:51:24.560077 1164 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.561726 1164 tablet_bootstrap.cc:654] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.562217 1186 ts_tablet_manager.cc:1918] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.562361 1186 log.cc:1198] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/229da341da6d4b88853afcad32af3904
I20250112 20:51:24.562798 1186 ts_tablet_manager.cc:1939] T 229da341da6d4b88853afcad32af3904 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.562876 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_68881a1762874e3bbc0ca6b8cf213276"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.563493 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68881a1762874e3bbc0ca6b8cf213276 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.565800 1186 tablet_replica.cc:331] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.566022 1186 raft_consensus.cc:2238] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.566196 1186 raft_consensus.cc:2267] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.567695 1186 ts_tablet_manager.cc:1905] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.569582 1164 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.569737 1164 ts_tablet_manager.cc:1397] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250112 20:51:24.570449 1164 raft_consensus.cc:357] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.570683 1164 raft_consensus.cc:383] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.570767 1164 raft_consensus.cc:738] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.570966 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet adb0973b4ae8457b888d614682f52b62
I20250112 20:51:24.571534 455 catalog_manager.cc:6596] ReplaceTablet: tablet adb0973b4ae8457b888d614682f52b62 deleted and replaced by tablet 719b906973a8490898d6befce49910d3
I20250112 20:51:24.573486 1164 consensus_queue.cc:260] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.576301 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 313a5c9905bc445893f1ffa8752ca589
I20250112 20:51:24.579614 1164 ts_tablet_manager.cc:1428] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:24.580288 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1" } modify_external_catalogs: true
I20250112 20:51:24.580492 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1" } modify_external_catalogs: true
I20250112 20:51:24.581182 1164 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.583758 1186 ts_tablet_manager.cc:1918] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.583967 1186 log.cc:1198] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5f125b1cc50643709f8666847c587266
I20250112 20:51:24.583987 1164 tablet_bootstrap.cc:654] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.584424 1186 ts_tablet_manager.cc:1939] T 5f125b1cc50643709f8666847c587266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.588060 1164 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.588198 1164 ts_tablet_manager.cc:1397] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:24.588097 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3d5bec5466fb4f71bbe9b86ec73f874f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.588596 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d5bec5466fb4f71bbe9b86ec73f874f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.588963 1164 raft_consensus.cc:357] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.589166 1164 raft_consensus.cc:383] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.589260 1164 raft_consensus.cc:738] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.590044 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_015060e7afea41ba852c420195188ddf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.590456 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_015060e7afea41ba852c420195188ddf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.591573 454 master_service.cc:946] ReplaceTablet: received request to replace tablet b07cef7306074ba3b7f8606bdfa6204a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.591948 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b862bcf0b0ed4f828f595e4e4e1d124e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.592340 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b862bcf0b0ed4f828f595e4e4e1d124e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.593093 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fb2be6ce07824fe6a27363d4d6ff0106"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.593698 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2be6ce07824fe6a27363d4d6ff0106 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.594502 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3a97fd937df444b7a25d357551f5cd85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.595114 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a97fd937df444b7a25d357551f5cd85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.595126 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_04f9f9491e5f49ec9660427758223e26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:24.595813 1164 consensus_queue.cc:260] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:24.596097 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04f9f9491e5f49ec9660427758223e26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.596724 753 tablet_service.cc:1467] Processing CreateTablet for tablet a48df2ec556d43a3b22129b1177ebf74 (DEFAULT_TABLE table=default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.597796 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48df2ec556d43a3b22129b1177ebf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.602519 1027 tablet_service.cc:1467] Processing CreateTablet for tablet a48df2ec556d43a3b22129b1177ebf74 (DEFAULT_TABLE table=default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.602993 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48df2ec556d43a3b22129b1177ebf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.604154 1164 ts_tablet_manager.cc:1428] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250112 20:51:24.605767 1164 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.612808 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.613147 893 tablet_service.cc:1467] Processing CreateTablet for tablet 313a5c9905bc445893f1ffa8752ca589 (DEFAULT_TABLE table=default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.613540 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313a5c9905bc445893f1ffa8752ca589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.614076 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.614457 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad0df4285db4bb6954f3e4c8d858115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.617012 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.617267 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ab78028b7ec14798be944daebc738998 with delete_type TABLET_DATA_DELETED (replaced by tablet eb1dffbe43ba4eeab7ee12d901b62ffb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.617656 891 tablet_service.cc:1467] Processing CreateTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb (DEFAULT_TABLE table=default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.618053 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1dffbe43ba4eeab7ee12d901b62ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.618093 892 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (replaced by tablet 719b906973a8490898d6befce49910d3) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.618223 885 tablet_service.cc:1467] Processing CreateTablet for tablet a48df2ec556d43a3b22129b1177ebf74 (DEFAULT_TABLE table=default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.618566 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48df2ec556d43a3b22129b1177ebf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.620937 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.621209 1169 raft_consensus.cc:491] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.621358 1169 raft_consensus.cc:513] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.622035 1169 leader_election.cc:290] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:24.622963 710 leader_election.cc:343] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.624313 751 tablet_service.cc:1467] Processing CreateTablet for tablet 313a5c9905bc445893f1ffa8752ca589 (DEFAULT_TABLE table=default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.624688 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313a5c9905bc445893f1ffa8752ca589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.624935 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.625146 750 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.625303 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ab78028b7ec14798be944daebc738998 with delete_type TABLET_DATA_DELETED (replaced by tablet eb1dffbe43ba4eeab7ee12d901b62ffb) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.625607 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.625913 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad0df4285db4bb6954f3e4c8d858115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.629025 749 tablet_service.cc:1467] Processing CreateTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb (DEFAULT_TABLE table=default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.629328 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1dffbe43ba4eeab7ee12d901b62ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.630156 1184 ts_tablet_manager.cc:1918] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.630293 1184 log.cc:1198] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.630707 1184 ts_tablet_manager.cc:1939] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.631348 888 tablet_service.cc:1467] Processing CreateTablet for tablet 382aefb4a5e34999b560a9237839b380 (DEFAULT_TABLE table=default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.631705 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 382aefb4a5e34999b560a9237839b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.632896 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.633149 748 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (replaced by tablet 719b906973a8490898d6befce49910d3) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.637465 1185 ts_tablet_manager.cc:1905] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.637986 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 313a5c9905bc445893f1ffa8752ca589 (DEFAULT_TABLE table=default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.638155 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.638988 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (replaced by tablet 719b906973a8490898d6befce49910d3) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.638263 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ab78028b7ec14798be944daebc738998 with delete_type TABLET_DATA_DELETED (replaced by tablet eb1dffbe43ba4eeab7ee12d901b62ffb) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.638531 1020 tablet_service.cc:1467] Processing CreateTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb (DEFAULT_TABLE table=default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:24.639344 710 leader_election.cc:343] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.639478 710 leader_election.cc:304] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:24.639537 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1dffbe43ba4eeab7ee12d901b62ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.639990 1169 raft_consensus.cc:2743] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:24.640444 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 382aefb4a5e34999b560a9237839b380 (DEFAULT_TABLE table=default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.640821 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 382aefb4a5e34999b560a9237839b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.645483 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.645715 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.646560 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]) successfully deleted
I20250112 20:51:24.638859 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.647399 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9898e024e23f4eb2ad38b7953dbd05d1 (table default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]) successfully deleted
W20250112 20:51:24.647657 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9fbcad11f1934759804e51554cd05e3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fbcad11f1934759804e51554cd05e3d already in progress: creating tablet
I20250112 20:51:24.647809 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (table default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
W20250112 20:51:24.647998 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6f79679bd64041e69e8c15369ec4acfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f79679bd64041e69e8c15369ec4acfb already in progress: creating tablet
I20250112 20:51:24.648142 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]) successfully deleted
I20250112 20:51:24.648293 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5f125b1cc50643709f8666847c587266 (table default.Table_ae6a03149fcb4b848a77bb2237a314ac [id=cac1b3ea76174bc5b68763819ad2b2e1]) successfully deleted
W20250112 20:51:24.648602 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e482a2543cdf4020a03f131c980e7928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e482a2543cdf4020a03f131c980e7928 already in progress: creating tablet
W20250112 20:51:24.648826 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e3a895feb0504d39ba8f93b25e2e032d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3a895feb0504d39ba8f93b25e2e032d already in progress: creating tablet
W20250112 20:51:24.649020 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ab78028b7ec14798be944daebc738998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab78028b7ec14798be944daebc738998 already in progress: creating tablet
W20250112 20:51:24.649233 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet adb0973b4ae8457b888d614682f52b62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adb0973b4ae8457b888d614682f52b62 already in progress: creating tablet
W20250112 20:51:24.649433 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 24abf89a9b2542d7bac2f7499698df3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24abf89a9b2542d7bac2f7499698df3a already in progress: creating tablet
I20250112 20:51:24.650321 1130 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:24.650480 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (table default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:24.650556 1130 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:24.653263 1185 ts_tablet_manager.cc:1918] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.653393 1185 log.cc:1198] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9898e024e23f4eb2ad38b7953dbd05d1
I20250112 20:51:24.655697 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.656070 1178 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.656236 1178 ts_tablet_manager.cc:1397] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.251s user 0.103s sys 0.107s
I20250112 20:51:24.657107 1178 raft_consensus.cc:357] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.657330 1178 raft_consensus.cc:383] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.657392 1178 raft_consensus.cc:738] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.657666 1178 consensus_queue.cc:260] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.638525 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.659356 1178 ts_tablet_manager.cc:1428] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:24.659617 1178 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.660051 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad0df4285db4bb6954f3e4c8d858115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.661284 753 tablet_service.cc:1467] Processing CreateTablet for tablet 382aefb4a5e34999b560a9237839b380 (DEFAULT_TABLE table=default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.661737 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313a5c9905bc445893f1ffa8752ca589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.662720 1178 tablet_bootstrap.cc:654] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.668264 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 382aefb4a5e34999b560a9237839b380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.670390 1185 ts_tablet_manager.cc:1939] T 9898e024e23f4eb2ad38b7953dbd05d1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.673437 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9898e024e23f4eb2ad38b7953dbd05d1 (table default.Table_d32f4745317e406797bf1b50c1e33102 [id=7f2daf8457654abbb137a538b38b124f]) successfully deleted
W20250112 20:51:24.673725 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6f79679bd64041e69e8c15369ec4acfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f79679bd64041e69e8c15369ec4acfb already in progress: creating tablet
W20250112 20:51:24.673949 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9fbcad11f1934759804e51554cd05e3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fbcad11f1934759804e51554cd05e3d already in progress: creating tablet
I20250112 20:51:24.674255 1164 tablet_bootstrap.cc:654] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.680377 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b07cef7306074ba3b7f8606bdfa6204a
I20250112 20:51:24.680632 1163 tablet_bootstrap.cc:492] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.680764 1163 ts_tablet_manager.cc:1397] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.404s user 0.089s sys 0.141s
I20250112 20:51:24.680891 454 catalog_manager.cc:6596] ReplaceTablet: tablet b07cef7306074ba3b7f8606bdfa6204a deleted and replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:24.681551 1163 raft_consensus.cc:357] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.681771 1163 raft_consensus.cc:383] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.681867 1163 raft_consensus.cc:738] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.682097 1163 consensus_queue.cc:260] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.682317 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d38bb6feb1d44f2fbff02b9530af3908" } new_table_name: "default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3" modify_external_catalogs: true
I20250112 20:51:24.682582 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.682647 1163 ts_tablet_manager.cc:1428] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:24.682807 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.682955 1163 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:24.683647 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b07cef7306074ba3b7f8606bdfa6204a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b07cef7306074ba3b7f8606bdfa6204a already in progress: creating tablet
I20250112 20:51:24.684446 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ce63edabd93a42389a8c93afb7f4cf00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:24.684614 1163 tablet_bootstrap.cc:654] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.684828 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_79cfd73169d6487388af1340e2b19a9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.684942 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce63edabd93a42389a8c93afb7f4cf00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:24.685302 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79cfd73169d6487388af1340e2b19a9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.686569 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_edf0cd8097e1425a86a5ef68749d2a6c" } new_table_name: "default.Table_a267a3d65a5849dca2db5fbd6f68c182" modify_external_catalogs: true
I20250112 20:51:24.688237 1185 tablet_replica.cc:331] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.688437 1185 raft_consensus.cc:2238] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.688588 1185 raft_consensus.cc:2267] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.689154 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 719b906973a8490898d6befce49910d3
I20250112 20:51:24.689323 1185 ts_tablet_manager.cc:1905] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.693248 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_78489ea1efe84eca854de9419d66b739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:24.693850 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78489ea1efe84eca854de9419d66b739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:24.696058 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 with delete_type TABLET_DATA_DELETED (replaced by tablet b07cef7306074ba3b7f8606bdfa6204a) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.696461 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 229da341da6d4b88853afcad32af3904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:23 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.700600 1185 ts_tablet_manager.cc:1918] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.703243 1184 tablet_replica.cc:331] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.703526 1184 raft_consensus.cc:2238] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.703722 1184 raft_consensus.cc:2267] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:24.706328 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e482a2543cdf4020a03f131c980e7928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e482a2543cdf4020a03f131c980e7928 already in progress: creating tablet
I20250112 20:51:24.700773 1185 log.cc:1198] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.706873 1185 ts_tablet_manager.cc:1939] T ad27af9a5df74c5a86bc7b0b05d9f8f3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:24.707099 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 313a5c9905bc445893f1ffa8752ca589 because the tablet was not found. No further retry: Not found: Tablet not found: 313a5c9905bc445893f1ffa8752ca589
W20250112 20:51:24.707486 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 24abf89a9b2542d7bac2f7499698df3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24abf89a9b2542d7bac2f7499698df3a already in progress: creating tablet
I20250112 20:51:24.707810 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 (table default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
W20250112 20:51:24.708055 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e3a895feb0504d39ba8f93b25e2e032d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3a895feb0504d39ba8f93b25e2e032d already in progress: creating tablet
I20250112 20:51:24.708412 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_886e8c220c4c495c9de8c57e28750a64" } new_table_name: "default.Table_dca6dbc93bff4fccaf0c975522fa3f74" modify_external_catalogs: true
I20250112 20:51:24.709864 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:24.710672 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_59b3292413f941d1a834d8dc55a640bc" } new_table_name: "default.Table_5bfc59824f23459cade39fdb87c63491" modify_external_catalogs: true
I20250112 20:51:24.710951 893 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.711262 1185 tablet_replica.cc:331] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.711422 1185 raft_consensus.cc:2238] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.711591 1185 raft_consensus.cc:2267] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:24.711604 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet df9fcf5bc7584cdcb17b19fc69802b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9fcf5bc7584cdcb17b19fc69802b84 already in progress: creating tablet
I20250112 20:51:24.711773 756 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.711962 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.712780 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.713909 1185 ts_tablet_manager.cc:1905] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.718487 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:24.719154 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 313a5c9905bc445893f1ffa8752ca589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 313a5c9905bc445893f1ffa8752ca589 already in progress: creating tablet
I20250112 20:51:24.721048 1163 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.721233 1163 ts_tablet_manager.cc:1397] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.003s
I20250112 20:51:24.722008 1163 raft_consensus.cc:357] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.722287 1163 raft_consensus.cc:383] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.722404 1163 raft_consensus.cc:738] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.722712 1163 consensus_queue.cc:260] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.723259 1163 ts_tablet_manager.cc:1428] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:24.723604 1163 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.725332 1163 tablet_bootstrap.cc:654] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:24.729777 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7df227e3389841259dc1420ec800e1c1 consensus state has no leader
I20250112 20:51:24.744853 1184 ts_tablet_manager.cc:1905] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.745231 1185 ts_tablet_manager.cc:1918] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.745380 1185 log.cc:1198] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e3a895feb0504d39ba8f93b25e2e032d
I20250112 20:51:24.745895 1185 ts_tablet_manager.cc:1939] T e3a895feb0504d39ba8f93b25e2e032d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.747180 1185 tablet_replica.cc:331] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.747226 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e3a895feb0504d39ba8f93b25e2e032d (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:24.747372 1185 raft_consensus.cc:2238] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.747567 1185 raft_consensus.cc:2267] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.747752 1163 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.747960 1163 ts_tablet_manager.cc:1397] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.003s
I20250112 20:51:24.748754 1185 ts_tablet_manager.cc:1905] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.748728 1163 raft_consensus.cc:357] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.749058 1163 raft_consensus.cc:383] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.749204 1163 raft_consensus.cc:738] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.749450 1163 consensus_queue.cc:260] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.752506 1163 ts_tablet_manager.cc:1428] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20250112 20:51:24.753820 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 consensus state has no leader
I20250112 20:51:24.753517 1164 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.755085 1164 ts_tablet_manager.cc:1397] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.011s
I20250112 20:51:24.755856 1164 raft_consensus.cc:357] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.756104 1164 raft_consensus.cc:383] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.756174 1164 raft_consensus.cc:738] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.756613 1164 consensus_queue.cc:260] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.757149 1164 ts_tablet_manager.cc:1428] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:24.757609 1164 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.759902 1185 ts_tablet_manager.cc:1918] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.760078 1185 log.cc:1198] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e482a2543cdf4020a03f131c980e7928
I20250112 20:51:24.760637 1185 ts_tablet_manager.cc:1939] T e482a2543cdf4020a03f131c980e7928 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.760751 1163 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.761590 1185 tablet_replica.cc:331] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.761812 1185 raft_consensus.cc:2238] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:24.762274 1185 pending_rounds.cc:62] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:24.762243 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e482a2543cdf4020a03f131c980e7928 (table default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]) successfully deleted
I20250112 20:51:24.762527 1185 pending_rounds.cc:69] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584985724104704 op_type: NO_OP noop_request { }
I20250112 20:51:24.762780 1185 raft_consensus.cc:2883] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:24.762955 1185 raft_consensus.cc:2267] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.762979 1163 tablet_bootstrap.cc:654] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.769053 1164 tablet_bootstrap.cc:654] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:24.769625 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e040383f6e2a4ccf9614fd2d119a3cb4 consensus state has no leader
I20250112 20:51:24.770653 1178 tablet_bootstrap.cc:492] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.770788 1178 ts_tablet_manager.cc:1397] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.111s user 0.029s sys 0.060s
I20250112 20:51:24.771466 1178 raft_consensus.cc:357] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.771656 1178 raft_consensus.cc:383] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.771729 1178 raft_consensus.cc:738] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.771955 1178 consensus_queue.cc:260] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:24.760389 1184 ts_tablet_manager.cc:1918] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.772948 1184 log.cc:1198] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ab78028b7ec14798be944daebc738998
I20250112 20:51:24.773339 1184 ts_tablet_manager.cc:1939] T ab78028b7ec14798be944daebc738998 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.774371 1184 tablet_replica.cc:331] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.774757 1184 raft_consensus.cc:2238] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:24.774765 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ab78028b7ec14798be944daebc738998 (table default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
W20250112 20:51:24.775048 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 because the tablet was not found. No further retry: Not found: Tablet not found: ad27af9a5df74c5a86bc7b0b05d9f8f3
I20250112 20:51:24.775161 1184 pending_rounds.cc:62] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:24.775336 1184 pending_rounds.cc:69] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584985698934784 op_type: NO_OP noop_request { }
W20250112 20:51:24.775478 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30e53e67076c4a5a89f1e93fc168a997 consensus state has no leader
I20250112 20:51:24.775532 1184 raft_consensus.cc:2883] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:24.775635 1184 raft_consensus.cc:2267] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.775655 1185 ts_tablet_manager.cc:1905] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:24.776101 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e814b7a9b4cb4336b62b7cd7e35461b6 consensus state has no leader
I20250112 20:51:24.777120 1184 ts_tablet_manager.cc:1905] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.777460 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]). Expected version 2 got 0
I20250112 20:51:24.778275 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]). Expected version 2 got 0
I20250112 20:51:24.778630 1178 ts_tablet_manager.cc:1428] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:24.779706 448 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]). Expected version 2 got 0
I20250112 20:51:24.782603 756 tablet_service.cc:1467] Processing CreateTablet for tablet 719b906973a8490898d6befce49910d3 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.782874 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 with delete_type TABLET_DATA_DELETED (replaced by tablet b07cef7306074ba3b7f8606bdfa6204a) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.783092 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.783300 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.783741 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b906973a8490898d6befce49910d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.786026 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]). Expected version 2 got 0
I20250112 20:51:24.787377 1178 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.790684 888 tablet_service.cc:1514] Processing DeleteTablet for tablet ab78028b7ec14798be944daebc738998 with delete_type TABLET_DATA_DELETED (replaced by tablet eb1dffbe43ba4eeab7ee12d901b62ffb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.790968 1186 tablet_replica.cc:331] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.790999 1178 tablet_bootstrap.cc:654] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.791134 1186 raft_consensus.cc:2238] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.791338 1186 raft_consensus.cc:2267] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.787806 1185 ts_tablet_manager.cc:1918] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:24.791996 1185 log.cc:1198] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ab78028b7ec14798be944daebc738998
I20250112 20:51:24.792346 1185 ts_tablet_manager.cc:1939] T ab78028b7ec14798be944daebc738998 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.792989 1186 ts_tablet_manager.cc:1905] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.793334 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.793563 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:24.794308 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ab78028b7ec14798be944daebc738998 (table default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
W20250112 20:51:24.794554 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 24abf89a9b2542d7bac2f7499698df3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24abf89a9b2542d7bac2f7499698df3a already in progress: creating tablet
W20250112 20:51:24.794873 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 313a5c9905bc445893f1ffa8752ca589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 313a5c9905bc445893f1ffa8752ca589 already in progress: creating tablet
W20250112 20:51:24.795348 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b07cef7306074ba3b7f8606bdfa6204a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b07cef7306074ba3b7f8606bdfa6204a already in progress: creating tablet
W20250112 20:51:24.795524 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet df9fcf5bc7584cdcb17b19fc69802b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9fcf5bc7584cdcb17b19fc69802b84 already in progress: creating tablet
W20250112 20:51:24.795703 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ad27af9a5df74c5a86bc7b0b05d9f8f3 because the tablet was not found. No further retry: Not found: Tablet not found: ad27af9a5df74c5a86bc7b0b05d9f8f3
W20250112 20:51:24.795876 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e3a895feb0504d39ba8f93b25e2e032d because the tablet was not found. No further retry: Not found: Tablet not found: e3a895feb0504d39ba8f93b25e2e032d
W20250112 20:51:24.796072 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e482a2543cdf4020a03f131c980e7928 because the tablet was not found. No further retry: Not found: Tablet not found: e482a2543cdf4020a03f131c980e7928
I20250112 20:51:24.796516 1184 ts_tablet_manager.cc:1918] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:24.796665 1184 log.cc:1198] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/229da341da6d4b88853afcad32af3904
I20250112 20:51:24.796917 888 tablet_service.cc:1467] Processing CreateTablet for tablet 719b906973a8490898d6befce49910d3 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.797137 1184 ts_tablet_manager.cc:1939] T 229da341da6d4b88853afcad32af3904 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.797140 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 719b906973a8490898d6befce49910d3 (DEFAULT_TABLE table=default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.797539 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b906973a8490898d6befce49910d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.798206 1178 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.798349 1178 ts_tablet_manager.cc:1397] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250112 20:51:24.799149 1178 raft_consensus.cc:357] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.799365 1178 raft_consensus.cc:383] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.799432 1178 raft_consensus.cc:738] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.799657 1178 consensus_queue.cc:260] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.800139 1178 ts_tablet_manager.cc:1428] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:24.800437 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719b906973a8490898d6befce49910d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.800490 1178 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.801548 1163 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.801779 1163 ts_tablet_manager.cc:1397] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250112 20:51:24.802234 1178 tablet_bootstrap.cc:654] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.802688 1163 raft_consensus.cc:357] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.802899 1163 raft_consensus.cc:383] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.802986 1163 raft_consensus.cc:738] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.806437 1163 consensus_queue.cc:260] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.808676 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.810382 1163 ts_tablet_manager.cc:1428] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:24.811794 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 229da341da6d4b88853afcad32af3904 (table default.Table_f3764d3dcba84aaa8597fa926a0dbe12 [id=d6b47c166bd447459187b6ac2905d26d]) successfully deleted
I20250112 20:51:24.812389 1163 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.812538 1184 tablet_replica.cc:331] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.812753 1184 raft_consensus.cc:2238] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.813006 1184 raft_consensus.cc:2267] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:24.813119 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b07cef7306074ba3b7f8606bdfa6204a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b07cef7306074ba3b7f8606bdfa6204a already in progress: creating tablet
W20250112 20:51:24.813378 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet df9fcf5bc7584cdcb17b19fc69802b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9fcf5bc7584cdcb17b19fc69802b84 already in progress: creating tablet
I20250112 20:51:24.813908 1184 ts_tablet_manager.cc:1905] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.813992 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_954436533cfc462dbffed1710e7920ca" } modify_external_catalogs: true
I20250112 20:51:24.814222 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_954436533cfc462dbffed1710e7920ca" } modify_external_catalogs: true
I20250112 20:51:24.816341 1163 tablet_bootstrap.cc:654] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.818205 1164 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.818399 1164 ts_tablet_manager.cc:1397] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.006s
I20250112 20:51:24.819213 1164 raft_consensus.cc:357] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.819492 1164 raft_consensus.cc:383] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.819640 1164 raft_consensus.cc:738] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.819767 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b862bcf0b0ed4f828f595e4e4e1d124e" } new_table_name: "default.Table_a8f2b2f1a79c4ad6a6defba7d45b05c7" modify_external_catalogs: true
I20250112 20:51:24.819953 1164 consensus_queue.cc:260] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.820730 1164 ts_tablet_manager.cc:1428] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:24.821033 1164 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.821427 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c9327af43e4f40429a836542ade34ab7" } modify_external_catalogs: true
I20250112 20:51:24.821606 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c9327af43e4f40429a836542ade34ab7" } modify_external_catalogs: true
I20250112 20:51:24.822108 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.822640 1164 tablet_bootstrap.cc:654] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.823845 446 master_service.cc:946] ReplaceTablet: received request to replace tablet adb0973b4ae8457b888d614682f52b62 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.826290 1184 ts_tablet_manager.cc:1918] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.826395 1184 log.cc:1198] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/adb0973b4ae8457b888d614682f52b62
I20250112 20:51:24.826736 1184 ts_tablet_manager.cc:1939] T adb0973b4ae8457b888d614682f52b62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.827518 1184 tablet_replica.cc:331] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:24.827723 1184 raft_consensus.cc:2238] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.827867 1184 raft_consensus.cc:2267] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.828568 1184 ts_tablet_manager.cc:1905] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.830893 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 7df227e3389841259dc1420ec800e1c1 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.830946 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet adb0973b4ae8457b888d614682f52b62 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
I20250112 20:51:24.831198 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 87756828be274286a3bb674040727828 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:24.831377 756 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.831676 1185 tablet_replica.cc:331] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.831914 1185 raft_consensus.cc:2238] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.832093 1185 raft_consensus.cc:2267] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.832780 1185 ts_tablet_manager.cc:1905] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.835847 1186 ts_tablet_manager.cc:1918] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.838593 1185 ts_tablet_manager.cc:1918] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.838707 1185 log.cc:1198] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/adb0973b4ae8457b888d614682f52b62
I20250112 20:51:24.839054 1185 ts_tablet_manager.cc:1939] T adb0973b4ae8457b888d614682f52b62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:24.839195 1186 log.cc:1198] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ab78028b7ec14798be944daebc738998
I20250112 20:51:24.839643 1186 ts_tablet_manager.cc:1939] T ab78028b7ec14798be944daebc738998 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.840051 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet adb0973b4ae8457b888d614682f52b62 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
I20250112 20:51:24.840667 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ab78028b7ec14798be944daebc738998 (table default.Table_6c0fcd020f404207804bc1fd94676779 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
I20250112 20:51:24.841017 1186 tablet_replica.cc:331] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.841243 1186 raft_consensus.cc:2238] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.841449 1186 raft_consensus.cc:2267] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.842300 1184 ts_tablet_manager.cc:1918] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.842437 1184 log.cc:1198] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:24.842734 1184 ts_tablet_manager.cc:1939] T 9fbcad11f1934759804e51554cd05e3d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:24.842751 756 tablet_service.cc:1467] Processing CreateTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 (DEFAULT_TABLE table=default.Table_78489ea1efe84eca854de9419d66b739 [id=2fd5583e6e114bcdbdbc723643735fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.843133 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d616a8ad9ad4777b44deb77bc321d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.843474 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 (DEFAULT_TABLE table=default.Table_78489ea1efe84eca854de9419d66b739 [id=2fd5583e6e114bcdbdbc723643735fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.843806 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d616a8ad9ad4777b44deb77bc321d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.849323 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 (DEFAULT_TABLE table=default.Table_78489ea1efe84eca854de9419d66b739 [id=2fd5583e6e114bcdbdbc723643735fd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.849772 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d616a8ad9ad4777b44deb77bc321d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.850833 1186 ts_tablet_manager.cc:1905] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.851075 1178 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.851234 1178 ts_tablet_manager.cc:1397] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.051s user 0.021s sys 0.024s
I20250112 20:51:24.852116 1178 raft_consensus.cc:357] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.852373 1178 raft_consensus.cc:383] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.852479 1178 raft_consensus.cc:738] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.852715 1178 consensus_queue.cc:260] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.853402 1178 ts_tablet_manager.cc:1428] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:24.853407 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--webserver_interface=127.0.104.61
--webserver_port=39277
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 20:51:24.853705 1178 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.855295 1178 tablet_bootstrap.cc:654] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.859720 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9fbcad11f1934759804e51554cd05e3d (table default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]) successfully deleted
I20250112 20:51:24.859715 748 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 (DEFAULT_TABLE table=default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.860183 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a8468369af418ea4a1b0a681f8ff74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.860877 1164 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:24.861050 1164 ts_tablet_manager.cc:1397] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250112 20:51:24.861105 753 tablet_service.cc:1467] Processing CreateTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 (DEFAULT_TABLE table=default.Table_79cfd73169d6487388af1340e2b19a9e [id=7bde50a83284433daf462fe09db9061c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.861481 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9aa4f42f89455cba7c72e7a49c6a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.861783 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (DEFAULT_TABLE table=default.Table_ce63edabd93a42389a8c93afb7f4cf00 [id=9448c50d601f435e81de18e95f86efb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.861848 1164 raft_consensus.cc:357] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.862082 1164 raft_consensus.cc:383] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.862134 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f012db68caf4d7d86afa6deb9f2ffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.862171 1164 raft_consensus.cc:738] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.862419 1164 consensus_queue.cc:260] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.862919 1164 ts_tablet_manager.cc:1428] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:24.863252 1164 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:24.865108 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4390c60d6c724190a216ef9c961b9592 (DEFAULT_TABLE table=default.Table_b862bcf0b0ed4f828f595e4e4e1d124e [id=2ee1f97fef0f4e2b83932132d0936504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.865088 751 tablet_service.cc:1467] Processing CreateTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.865232 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (DEFAULT_TABLE table=default.Table_3a97fd937df444b7a25d357551f5cd85 [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.865398 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4390c60d6c724190a216ef9c961b9592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.865413 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed45eb5cbd2e489c83c3d343ca1adcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.865607 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.865883 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fc63526f932841509277f4726303c801 (DEFAULT_TABLE table=default.Table_015060e7afea41ba852c420195188ddf [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.866200 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc63526f932841509277f4726303c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.866868 1022 tablet_service.cc:1467] Processing CreateTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 (DEFAULT_TABLE table=default.Table_79cfd73169d6487388af1340e2b19a9e [id=7bde50a83284433daf462fe09db9061c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.867297 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9aa4f42f89455cba7c72e7a49c6a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.869053 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 296703eedc6d459880a74b38e642bceb (DEFAULT_TABLE table=default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.869037 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4390c60d6c724190a216ef9c961b9592 (DEFAULT_TABLE table=default.Table_b862bcf0b0ed4f828f595e4e4e1d124e [id=2ee1f97fef0f4e2b83932132d0936504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.869338 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4390c60d6c724190a216ef9c961b9592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.869344 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 296703eedc6d459880a74b38e642bceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.873116 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.873174 750 tablet_service.cc:1467] Processing CreateTablet for tablet 296703eedc6d459880a74b38e642bceb (DEFAULT_TABLE table=default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.873536 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 296703eedc6d459880a74b38e642bceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.873577 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed45eb5cbd2e489c83c3d343ca1adcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.877097 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 (DEFAULT_TABLE table=default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.877192 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (DEFAULT_TABLE table=default.Table_3a97fd937df444b7a25d357551f5cd85 [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.877516 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.877527 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a8468369af418ea4a1b0a681f8ff74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.881017 752 tablet_service.cc:1467] Processing CreateTablet for tablet fc63526f932841509277f4726303c801 (DEFAULT_TABLE table=default.Table_015060e7afea41ba852c420195188ddf [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.881381 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc63526f932841509277f4726303c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.883714 1181 raft_consensus.cc:491] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.883948 1181 raft_consensus.cc:513] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.886575 889 tablet_service.cc:1467] Processing CreateTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 (DEFAULT_TABLE table=default.Table_79cfd73169d6487388af1340e2b19a9e [id=7bde50a83284433daf462fe09db9061c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.886983 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9aa4f42f89455cba7c72e7a49c6a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.886951 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (DEFAULT_TABLE table=default.Table_ce63edabd93a42389a8c93afb7f4cf00 [id=9448c50d601f435e81de18e95f86efb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.887275 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f012db68caf4d7d86afa6deb9f2ffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.880621 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 (DEFAULT_TABLE table=default.Table_68881a1762874e3bbc0ca6b8cf213276 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.887945 883 tablet_service.cc:1467] Processing CreateTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 (DEFAULT_TABLE table=default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.888418 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b17628cf145b69e5fdde4131d9fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.888511 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea7336d13ea46748167e35e7e7e8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.889106 891 tablet_service.cc:1467] Processing CreateTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc (DEFAULT_TABLE table=default.Table_0715c21f34a74a34ba9baeb1f71414d3 [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.889426 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed45eb5cbd2e489c83c3d343ca1adcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.893047 886 tablet_service.cc:1467] Processing CreateTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 (DEFAULT_TABLE table=default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.893378 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a8468369af418ea4a1b0a681f8ff74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.897042 892 tablet_service.cc:1467] Processing CreateTablet for tablet 4390c60d6c724190a216ef9c961b9592 (DEFAULT_TABLE table=default.Table_b862bcf0b0ed4f828f595e4e4e1d124e [id=2ee1f97fef0f4e2b83932132d0936504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.897220 1181 leader_election.cc:290] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:24.897387 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4390c60d6c724190a216ef9c961b9592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.901067 884 tablet_service.cc:1467] Processing CreateTablet for tablet fc63526f932841509277f4726303c801 (DEFAULT_TABLE table=default.Table_015060e7afea41ba852c420195188ddf [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.901400 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc63526f932841509277f4726303c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.905020 887 tablet_service.cc:1467] Processing CreateTablet for tablet 296703eedc6d459880a74b38e642bceb (DEFAULT_TABLE table=default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.905318 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 296703eedc6d459880a74b38e642bceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.908947 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 (DEFAULT_TABLE table=default.Table_68881a1762874e3bbc0ca6b8cf213276 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.909289 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea7336d13ea46748167e35e7e7e8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.911079 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 (DEFAULT_TABLE table=default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.911441 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b17628cf145b69e5fdde4131d9fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.913086 1178 tablet_bootstrap.cc:492] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:24.913285 1178 ts_tablet_manager.cc:1397] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.060s user 0.022s sys 0.013s
I20250112 20:51:24.913219 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (DEFAULT_TABLE table=default.Table_3a97fd937df444b7a25d357551f5cd85 [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.913580 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.914247 1178 raft_consensus.cc:357] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.914551 1178 raft_consensus.cc:383] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.914640 1178 raft_consensus.cc:738] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.914929 1178 consensus_queue.cc:260] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.915692 1178 ts_tablet_manager.cc:1428] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:24.916025 1178 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:24.917690 1163 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:24.917822 1163 ts_tablet_manager.cc:1397] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.007s
I20250112 20:51:24.918041 1178 tablet_bootstrap.cc:654] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.918650 1163 raft_consensus.cc:357] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.918851 1163 raft_consensus.cc:383] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.918944 1163 raft_consensus.cc:738] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.919176 1163 consensus_queue.cc:260] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.919664 1163 ts_tablet_manager.cc:1428] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:24.919952 1163 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:24.922238 1163 tablet_bootstrap.cc:654] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.928658 1186 ts_tablet_manager.cc:1918] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:24.933529 1186 log.cc:1198] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e3a895feb0504d39ba8f93b25e2e032d
I20250112 20:51:24.934012 1186 ts_tablet_manager.cc:1939] T e3a895feb0504d39ba8f93b25e2e032d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:24.936100 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.950937 1186 tablet_replica.cc:331] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:24.951135 1186 raft_consensus.cc:2238] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.951315 1186 raft_consensus.cc:2267] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.952165 1186 ts_tablet_manager.cc:1905] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:24.954560 1164 tablet_bootstrap.cc:654] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.956539 1169 raft_consensus.cc:491] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.956736 1169 raft_consensus.cc:513] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:24.957711 1169 leader_election.cc:290] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:24.959213 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87756828be274286a3bb674040727828" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:24.959522 1048 raft_consensus.cc:2463] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:24.959853 754 tablet_service.cc:1467] Processing CreateTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 (DEFAULT_TABLE table=default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.960644 1168 raft_consensus.cc:491] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:24.960861 1168 raft_consensus.cc:513] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.961689 1168 leader_election.cc:290] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:24.962249 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e3a895feb0504d39ba8f93b25e2e032d (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:24.963613 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5bfc59824f23459cade39fdb87c63491" } modify_external_catalogs: true
I20250112 20:51:24.963877 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5bfc59824f23459cade39fdb87c63491" } modify_external_catalogs: true
I20250112 20:51:24.967494 710 leader_election.cc:304] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:24.968755 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b17628cf145b69e5fdde4131d9fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.973189 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:24.973531 776 raft_consensus.cc:2463] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:24.974354 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 (DEFAULT_TABLE table=default.Table_68881a1762874e3bbc0ca6b8cf213276 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.974838 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea7336d13ea46748167e35e7e7e8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.978739 983 leader_election.cc:304] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:24.979187 1168 raft_consensus.cc:2798] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:24.979307 1168 raft_consensus.cc:491] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:24.979452 1168 raft_consensus.cc:3054] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.983126 1168 raft_consensus.cc:513] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:24.984438 1168 leader_election.cc:290] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:24.984825 1169 raft_consensus.cc:2798] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:24.985826 1169 raft_consensus.cc:491] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:24.985945 1169 raft_consensus.cc:3054] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.987200 884 tablet_service.cc:1514] Processing DeleteTablet for tablet adb0973b4ae8457b888d614682f52b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:24.994189 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:24.994460 776 raft_consensus.cc:3054] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.996841 747 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:24.997182 1185 tablet_replica.cc:331] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:24.997110 756 tablet_service.cc:1467] Processing CreateTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (DEFAULT_TABLE table=default.Table_ce63edabd93a42389a8c93afb7f4cf00 [id=9448c50d601f435e81de18e95f86efb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:24.997371 1185 raft_consensus.cc:2238] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:24.997481 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f012db68caf4d7d86afa6deb9f2ffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.997545 1185 raft_consensus.cc:2267] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:24.998338 1185 ts_tablet_manager.cc:1905] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.002306 776 raft_consensus.cc:2463] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:25.002872 983 leader_election.cc:304] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:25.003373 1168 raft_consensus.cc:2798] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:25.006644 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4390c60d6c724190a216ef9c961b9592 consensus state has no leader
I20250112 20:51:25.020303 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5dd7d40d429f4aa08fedf80d8acb42a7
I20250112 20:51:25.021291 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7df227e3389841259dc1420ec800e1c1
I20250112 20:51:25.021790 446 catalog_manager.cc:6596] ReplaceTablet: tablet 7df227e3389841259dc1420ec800e1c1 deleted and replaced by tablet 616b8aa8fd62486c978c275712064f77
I20250112 20:51:25.022185 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8c7a94b2e9be4b5db3dfd6129b03cd25
I20250112 20:51:25.022629 455 catalog_manager.cc:6596] ReplaceTablet: tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 deleted and replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:25.023051 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet aa3bd4eb92114bdb98b923fda9192cd6
I20250112 20:51:25.023824 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 87756828be274286a3bb674040727828
I20250112 20:51:25.024323 454 catalog_manager.cc:6596] ReplaceTablet: tablet 87756828be274286a3bb674040727828 deleted and replaced by tablet 0871d4272e874a958174f96bfc0e1a95
I20250112 20:51:25.036909 1169 raft_consensus.cc:513] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.037915 1169 leader_election.cc:290] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.040292 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87756828be274286a3bb674040727828" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:25.040594 1048 raft_consensus.cc:3054] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.042970 1185 ts_tablet_manager.cc:1918] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.043123 1185 log.cc:1198] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:25.043517 1185 ts_tablet_manager.cc:1939] T df9fcf5bc7584cdcb17b19fc69802b84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:25.044377 1048 raft_consensus.cc:2463] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:25.053617 710 leader_election.cc:304] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:25.054106 1169 raft_consensus.cc:2798] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:25.064965 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e814b7a9b4cb4336b62b7cd7e35461b6
I20250112 20:51:25.071703 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f8ae4b86db714d6bba40875699ee81a8" } modify_external_catalogs: true
I20250112 20:51:25.072088 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f8ae4b86db714d6bba40875699ee81a8" } modify_external_catalogs: true
I20250112 20:51:25.074188 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_80526018642b4dd5882083cb240f243d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.075116 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80526018642b4dd5882083cb240f243d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.077551 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87756828be274286a3bb674040727828" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.077977 1181 raft_consensus.cc:491] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.078691 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1b2cd38a56934793bb10f3381448a3ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.079097 710 leader_election.cc:343] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:25.079375 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b2cd38a56934793bb10f3381448a3ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.079748 1181 raft_consensus.cc:513] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.080555 1181 leader_election.cc:290] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.080788 1181 raft_consensus.cc:491] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.080938 1181 raft_consensus.cc:513] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.081274 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_598b0b1981fc4153baa6e9b8b2521981"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.081704 1181 leader_election.cc:290] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:25.081842 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_598b0b1981fc4153baa6e9b8b2521981 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.083783 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_71341c6c973c4559949e989d69912a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.084511 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71341c6c973c4559949e989d69912a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.086149 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 30e53e67076c4a5a89f1e93fc168a997 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.086576 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87756828be274286a3bb674040727828" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:25.086814 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6e7ee1324dba4c9398f4b88928f27d86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.087391 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e7ee1324dba4c9398f4b88928f27d86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.115747 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 30e53e67076c4a5a89f1e93fc168a997
I20250112 20:51:25.115963 1186 ts_tablet_manager.cc:1918] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.116128 1186 log.cc:1198] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:25.116370 446 catalog_manager.cc:6596] ReplaceTablet: tablet 30e53e67076c4a5a89f1e93fc168a997 deleted and replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:25.116592 1186 ts_tablet_manager.cc:1939] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:25.119572 710 leader_election.cc:343] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.125509 1186 tablet_replica.cc:331] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.125741 1186 raft_consensus.cc:2238] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.125962 1186 raft_consensus.cc:2267] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.126818 1186 ts_tablet_manager.cc:1905] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.133492 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2feab5fd4f5b4753ab157456f04f7e70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.134045 1186 ts_tablet_manager.cc:1918] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.134184 1186 log.cc:1198] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/adb0973b4ae8457b888d614682f52b62
W20250112 20:51:25.134312 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2feab5fd4f5b4753ab157456f04f7e70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.134541 1186 ts_tablet_manager.cc:1939] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:25.136401 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3" } modify_external_catalogs: true
I20250112 20:51:25.136619 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3" } modify_external_catalogs: true
I20250112 20:51:25.137607 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_884652a61536446f903e8a58dac9590f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.138510 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:25.138723 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_884652a61536446f903e8a58dac9590f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.138976 774 raft_consensus.cc:2388] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
W20250112 20:51:25.139730 847 leader_election.cc:343] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: adb0973b4ae8457b888d614682f52b62
I20250112 20:51:25.140362 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbcad11f1934759804e51554cd05e3d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:25.154317 847 leader_election.cc:343] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.162981 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.163219 911 raft_consensus.cc:2463] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:25.163793 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:25.164036 911 raft_consensus.cc:3054] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.167376 911 raft_consensus.cc:2463] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:25.171026 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd7d40d429f4aa08fedf80d8acb42a7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:25.171856 847 leader_election.cc:343] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:25.172101 847 leader_election.cc:304] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:25.172643 847 raft_consensus.cc:2681] T 9fbcad11f1934759804e51554cd05e3d P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 20:51:25.173900 847 leader_election.cc:343] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: adb0973b4ae8457b888d614682f52b62
I20250112 20:51:25.174172 847 leader_election.cc:304] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:25.174391 847 raft_consensus.cc:2681] T adb0973b4ae8457b888d614682f52b62 P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:25.174582 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 382aefb4a5e34999b560a9237839b380
I20250112 20:51:25.179502 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2a522da7a096468db3231e0f9f8205bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.180270 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a522da7a096468db3231e0f9f8205bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.181941 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0715c21f34a74a34ba9baeb1f71414d3" } new_table_name: "default.Table_07853a7e990447d293c9fd0530206abf" modify_external_catalogs: true
I20250112 20:51:25.203675 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 87756828be274286a3bb674040727828 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.204707 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.205106 1186 tablet_replica.cc:331] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.205093 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.205327 1186 raft_consensus.cc:2238] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.205436 1185 tablet_replica.cc:331] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:25.205585 1186 raft_consensus.cc:2267] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.205669 1185 raft_consensus.cc:2238] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.205900 1185 raft_consensus.cc:2267] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.206204 1186 ts_tablet_manager.cc:1905] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.206512 1185 ts_tablet_manager.cc:1905] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.213421 1186 ts_tablet_manager.cc:1918] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.213550 1186 log.cc:1198] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5dd7d40d429f4aa08fedf80d8acb42a7
I20250112 20:51:25.213927 1186 ts_tablet_manager.cc:1939] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:25.215541 1185 ts_tablet_manager.cc:1918] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.215709 1185 log.cc:1198] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5dd7d40d429f4aa08fedf80d8acb42a7
I20250112 20:51:25.216168 1185 ts_tablet_manager.cc:1939] T 5dd7d40d429f4aa08fedf80d8acb42a7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:25.226219 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ea30f59545d4a81929c403c64374844" } new_table_name: "default.Table_088900cac1084a659ca9af91b6c0276b" modify_external_catalogs: true
I20250112 20:51:25.229514 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b0367c09c21647a4beea62b7dd6d3d19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.230404 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0367c09c21647a4beea62b7dd6d3d19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.231330 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4911eabcac2c4a8b9d4b08525597082a" } new_table_name: "default.Table_c5e95d792c48441db50a9e741756db91" modify_external_catalogs: true
I20250112 20:51:25.232077 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_68881a1762874e3bbc0ca6b8cf213276" } new_table_name: "default.Table_e5db37687d384be894f7f6239198cd82" modify_external_catalogs: true
I20250112 20:51:25.233093 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0fa5226c1b51464da6a1b54f22ef816e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.233858 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fa5226c1b51464da6a1b54f22ef816e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.235903 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 616b8aa8fd62486c978c275712064f77
I20250112 20:51:25.238215 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.238574 1184 tablet_replica.cc:331] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:25.239073 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.239921 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.256930 1163 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.257324 1163 ts_tablet_manager.cc:1397] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.337s user 0.105s sys 0.199s
I20250112 20:51:25.257263 1168 raft_consensus.cc:695] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:25.257956 1164 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.257860 1168 consensus_queue.cc:237] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.258121 1164 ts_tablet_manager.cc:1397] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.395s user 0.018s sys 0.075s
I20250112 20:51:25.258073 1163 raft_consensus.cc:357] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.258430 1163 raft_consensus.cc:383] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.258514 1163 raft_consensus.cc:738] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.258796 1163 consensus_queue.cc:260] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.259274 1163 ts_tablet_manager.cc:1428] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:25.259145 1164 raft_consensus.cc:357] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.259528 1164 raft_consensus.cc:383] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.259610 1163 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.259640 1164 raft_consensus.cc:738] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.259929 1164 consensus_queue.cc:260] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.260366 1169 raft_consensus.cc:695] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:25.260545 1164 ts_tablet_manager.cc:1428] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250112 20:51:25.260875 1169 consensus_queue.cc:237] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.261559 1163 tablet_bootstrap.cc:654] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.262516 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.263535 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.264483 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.268296 1164 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.272173 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.273793 1164 tablet_bootstrap.cc:654] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.273875 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.274834 753 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.276314 888 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.277957 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 87756828be274286a3bb674040727828 with delete_type TABLET_DATA_DELETED (replaced by tablet 0871d4272e874a958174f96bfc0e1a95) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.279049 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 87756828be274286a3bb674040727828 with delete_type TABLET_DATA_DELETED (replaced by tablet 0871d4272e874a958174f96bfc0e1a95) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.279968 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 87756828be274286a3bb674040727828 with delete_type TABLET_DATA_DELETED (replaced by tablet 0871d4272e874a958174f96bfc0e1a95) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.280308 1185 tablet_replica.cc:331] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:25.282454 1184 raft_consensus.cc:2238] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:25.283282 1184 pending_rounds.cc:62] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:25.283363 1184 pending_rounds.cc:69] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584989221453824 op_type: NO_OP noop_request { }
I20250112 20:51:25.283551 1184 raft_consensus.cc:2883] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:25.283668 1184 raft_consensus.cc:2267] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:25.285780 847 leader_election.cc:343] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.285957 847 leader_election.cc:304] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:25.286217 847 raft_consensus.cc:2681] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:25.288612 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.290198 1164 tablet_bootstrap.cc:492] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.290427 1164 ts_tablet_manager.cc:1397] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20250112 20:51:25.291746 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.291661 1164 raft_consensus.cc:357] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.291944 1164 raft_consensus.cc:383] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.292042 1164 raft_consensus.cc:738] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.292368 1164 consensus_queue.cc:260] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.292714 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.293210 1164 ts_tablet_manager.cc:1428] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250112 20:51:25.293933 1164 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.296521 1164 tablet_bootstrap.cc:654] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:25.298275 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7a57ea399194a45bc8990a029550440 consensus state has no leader
W20250112 20:51:25.298785 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed45eb5cbd2e489c83c3d343ca1adcfc consensus state has no leader
I20250112 20:51:25.306655 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 with delete_type TABLET_DATA_DELETED (replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.307332 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 with delete_type TABLET_DATA_DELETED (replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.308027 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 with delete_type TABLET_DATA_DELETED (replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.311681 1164 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.311868 1164 ts_tablet_manager.cc:1397] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20250112 20:51:25.311885 888 tablet_service.cc:1467] Processing CreateTablet for tablet 0871d4272e874a958174f96bfc0e1a95 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.312405 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0871d4272e874a958174f96bfc0e1a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.312851 1164 raft_consensus.cc:357] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.313123 1164 raft_consensus.cc:383] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.313237 1164 raft_consensus.cc:738] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.313532 1164 consensus_queue.cc:260] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.314389 1164 ts_tablet_manager.cc:1428] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:25.314743 1164 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.316815 1164 tablet_bootstrap.cc:654] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.323056 1184 ts_tablet_manager.cc:1905] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.331595 1184 ts_tablet_manager.cc:1918] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:25.331771 1184 log.cc:1198] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5dd7d40d429f4aa08fedf80d8acb42a7
I20250112 20:51:25.332273 1184 ts_tablet_manager.cc:1939] T 5dd7d40d429f4aa08fedf80d8acb42a7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:25.333683 1168 raft_consensus.cc:491] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.333863 1168 raft_consensus.cc:513] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.335244 1164 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.335206 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9fcf5bc7584cdcb17b19fc69802b84" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.335460 1164 ts_tablet_manager.cc:1397] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
W20250112 20:51:25.335928 983 leader_election.cc:343] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.336410 1164 raft_consensus.cc:357] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.336722 1164 raft_consensus.cc:383] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.336819 1164 raft_consensus.cc:738] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.337141 1164 consensus_queue.cc:260] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:25.337859 983 leader_election.cc:343] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:25.337980 1164 ts_tablet_manager.cc:1428] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:25.338006 983 leader_election.cc:304] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:25.338320 1164 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.339028 1226 raft_consensus.cc:2743] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.340165 1168 leader_election.cc:290] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:25.341526 1164 tablet_bootstrap.cc:654] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.347841 1185 raft_consensus.cc:2238] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:25.348819 1185 pending_rounds.cc:62] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:25.349001 1185 pending_rounds.cc:69] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584989231800320 op_type: NO_OP noop_request { }
I20250112 20:51:25.349215 1185 raft_consensus.cc:2883] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:25.349433 1185 raft_consensus.cc:2267] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.350601 1185 ts_tablet_manager.cc:1905] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.354115 1164 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.354250 1164 ts_tablet_manager.cc:1397] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250112 20:51:25.355264 1164 raft_consensus.cc:357] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.355523 1164 raft_consensus.cc:383] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.355618 1164 raft_consensus.cc:738] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.355671 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0871d4272e874a958174f96bfc0e1a95 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.355957 1164 consensus_queue.cc:260] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.356251 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0871d4272e874a958174f96bfc0e1a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.356599 1164 ts_tablet_manager.cc:1428] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:25.357906 1164 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.360008 1164 tablet_bootstrap.cc:654] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.371512 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0871d4272e874a958174f96bfc0e1a95 (DEFAULT_TABLE table=default.Table_4911eabcac2c4a8b9d4b08525597082a [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.372126 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0871d4272e874a958174f96bfc0e1a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.374259 1178 tablet_bootstrap.cc:492] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.374321 1164 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.374469 1164 ts_tablet_manager.cc:1397] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20250112 20:51:25.374603 1178 ts_tablet_manager.cc:1397] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.459s user 0.160s sys 0.223s
I20250112 20:51:25.375192 1164 raft_consensus.cc:357] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.375458 1164 raft_consensus.cc:383] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.375531 1164 raft_consensus.cc:738] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.375782 1164 consensus_queue.cc:260] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.376060 1178 raft_consensus.cc:357] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.376405 1178 raft_consensus.cc:383] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.377013 1164 ts_tablet_manager.cc:1428] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:25.377285 1164 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.376572 1178 raft_consensus.cc:738] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.379367 1178 consensus_queue.cc:260] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.379745 1164 tablet_bootstrap.cc:654] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.385350 1178 ts_tablet_manager.cc:1428] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:25.386384 1178 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.388584 1178 tablet_bootstrap.cc:654] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.389098 1164 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.389233 1164 ts_tablet_manager.cc:1397] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20250112 20:51:25.390223 1164 raft_consensus.cc:357] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.390424 1164 raft_consensus.cc:383] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.390514 1164 raft_consensus.cc:738] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.390722 1164 consensus_queue.cc:260] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.393864 1164 ts_tablet_manager.cc:1428] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:25.394788 1184 tablet_replica.cc:331] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:25.395084 1184 raft_consensus.cc:2238] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.395367 1184 raft_consensus.cc:2267] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.395911 1164 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.396132 1184 ts_tablet_manager.cc:1905] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.397478 1164 tablet_bootstrap.cc:654] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.401415 753 tablet_service.cc:1467] Processing CreateTablet for tablet 616b8aa8fd62486c978c275712064f77 (DEFAULT_TABLE table=default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.401911 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616b8aa8fd62486c978c275712064f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.402482 893 tablet_service.cc:1467] Processing CreateTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 (DEFAULT_TABLE table=default.Table_a267a3d65a5849dca2db5fbd6f68c182 [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.402518 752 tablet_service.cc:1467] Processing CreateTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 (DEFAULT_TABLE table=default.Table_a267a3d65a5849dca2db5fbd6f68c182 [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.402819 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953d2ec8a7564c0bbcf0166487dd1746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.403723 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 (DEFAULT_TABLE table=default.Table_a267a3d65a5849dca2db5fbd6f68c182 [id=8bd86656bb82483ea8cd34a80fa2b843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.404124 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953d2ec8a7564c0bbcf0166487dd1746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.405282 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.405426 888 tablet_service.cc:1467] Processing CreateTablet for tablet 616b8aa8fd62486c978c275712064f77 (DEFAULT_TABLE table=default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.406025 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616b8aa8fd62486c978c275712064f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.407034 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.408941 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.410989 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953d2ec8a7564c0bbcf0166487dd1746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.412245 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 616b8aa8fd62486c978c275712064f77 (DEFAULT_TABLE table=default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.412676 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616b8aa8fd62486c978c275712064f77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:25.419369 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0871d4272e874a958174f96bfc0e1a95 consensus state has no leader
I20250112 20:51:25.423950 1164 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.424121 1164 ts_tablet_manager.cc:1397] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
I20250112 20:51:25.425062 1164 raft_consensus.cc:357] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.425336 1164 raft_consensus.cc:383] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.425431 1164 raft_consensus.cc:738] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.425737 1164 consensus_queue.cc:260] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.429140 1164 ts_tablet_manager.cc:1428] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:25.431983 1164 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.433781 1164 tablet_bootstrap.cc:654] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.436430 1164 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.436566 1164 ts_tablet_manager.cc:1397] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:51:25.437332 1164 raft_consensus.cc:357] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.437527 1164 raft_consensus.cc:383] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.437615 1164 raft_consensus.cc:738] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.437831 1164 consensus_queue.cc:260] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:25.440980 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ea7336d13ea46748167e35e7e7e8c98 consensus state has no leader
I20250112 20:51:25.441074 1164 ts_tablet_manager.cc:1428] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:25.444504 1164 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.446065 1164 tablet_bootstrap.cc:654] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.449085 1164 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.449247 1164 ts_tablet_manager.cc:1397] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:25.449892 1164 raft_consensus.cc:357] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.450116 1164 raft_consensus.cc:383] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.450186 1164 raft_consensus.cc:738] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.450410 1164 consensus_queue.cc:260] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.453573 1164 ts_tablet_manager.cc:1428] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:25.456683 1164 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.458879 1164 tablet_bootstrap.cc:654] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.461277 1164 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.461412 1164 ts_tablet_manager.cc:1397] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:25.462100 1164 raft_consensus.cc:357] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.462297 1164 raft_consensus.cc:383] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.462383 1164 raft_consensus.cc:738] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.462582 1164 consensus_queue.cc:260] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.469376 1169 raft_consensus.cc:491] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.469535 1169 raft_consensus.cc:513] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.470140 1169 leader_election.cc:290] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:25.471021 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:25.471506 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.471681 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet df9fcf5bc7584cdcb17b19fc69802b84 (table default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1 [id=a578996c09484fb1a6b08eb55eff74d0]) successfully deleted
W20250112 20:51:25.472466 710 leader_election.cc:343] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.472628 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]) successfully deleted
W20250112 20:51:25.472975 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 already in progress: creating tablet
W20250112 20:51:25.473105 710 leader_election.cc:343] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.473222 710 leader_election.cc:304] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:25.473237 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet aa3bd4eb92114bdb98b923fda9192cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3bd4eb92114bdb98b923fda9192cd6 already in progress: creating tablet
I20250112 20:51:25.474344 1169 raft_consensus.cc:2743] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.476337 1178 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.476552 1178 ts_tablet_manager.cc:1397] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.001s
I20250112 20:51:25.478991 1164 ts_tablet_manager.cc:1428] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 20:51:25.482574 1164 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.484519 1164 tablet_bootstrap.cc:654] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.493472 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9fbcad11f1934759804e51554cd05e3d (table default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]) successfully deleted
I20250112 20:51:25.493809 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet adb0973b4ae8457b888d614682f52b62 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
I20250112 20:51:25.494138 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]) successfully deleted
W20250112 20:51:25.494503 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 already in progress: creating tablet
W20250112 20:51:25.494748 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet aa3bd4eb92114bdb98b923fda9192cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3bd4eb92114bdb98b923fda9192cd6 already in progress: creating tablet
W20250112 20:51:25.494973 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 87756828be274286a3bb674040727828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87756828be274286a3bb674040727828 already in progress: creating tablet
W20250112 20:51:25.495201 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e814b7a9b4cb4336b62b7cd7e35461b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e814b7a9b4cb4336b62b7cd7e35461b6 already in progress: creating tablet
W20250112 20:51:25.495431 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 30e53e67076c4a5a89f1e93fc168a997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30e53e67076c4a5a89f1e93fc168a997 already in progress: creating tablet
W20250112 20:51:25.495779 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 382aefb4a5e34999b560a9237839b380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 382aefb4a5e34999b560a9237839b380 already in progress: creating tablet
I20250112 20:51:25.496603 1164 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.496838 1164 ts_tablet_manager.cc:1397] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20250112 20:51:25.497709 1164 raft_consensus.cc:357] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.498000 1164 raft_consensus.cc:383] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.498121 1164 raft_consensus.cc:738] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.498448 1164 consensus_queue.cc:260] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.502439 1164 ts_tablet_manager.cc:1428] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:25.506086 1164 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.506984 1178 raft_consensus.cc:357] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.507426 1178 raft_consensus.cc:383] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.507534 1178 raft_consensus.cc:738] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.507899 1178 consensus_queue.cc:260] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.508105 1164 tablet_bootstrap.cc:654] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.521389 1164 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.521615 1164 ts_tablet_manager.cc:1397] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20250112 20:51:25.522418 1164 raft_consensus.cc:357] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.522745 1164 raft_consensus.cc:383] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.522953 1164 raft_consensus.cc:738] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.523294 1164 consensus_queue.cc:260] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.528257 1164 ts_tablet_manager.cc:1428] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250112 20:51:25.531574 1164 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.533222 1178 ts_tablet_manager.cc:1428] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.057s user 0.001s sys 0.003s
I20250112 20:51:25.534343 1164 tablet_bootstrap.cc:654] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.536352 1178 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.537962 1178 tablet_bootstrap.cc:654] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.539254 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:25.539865 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.541131 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.541399 1186 tablet_replica.cc:331] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.541566 1186 raft_consensus.cc:2238] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.541719 1186 raft_consensus.cc:2267] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.541914 1169 raft_consensus.cc:491] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.541853 1021 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1c76db0d64624bd43f0e538f0649d (DEFAULT_TABLE table=default.Table_884652a61536446f903e8a58dac9590f [id=5178ded07b844cf4abc1616c387b8585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.542052 1169 raft_consensus.cc:513] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.542284 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f1c76db0d64624bd43f0e538f0649d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.542474 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_78489ea1efe84eca854de9419d66b739" } new_table_name: "default.Table_29dce4195b10489a9cc62c236b93b8f7" modify_external_catalogs: true
I20250112 20:51:25.542639 1169 leader_election.cc:290] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:25.543009 753 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1c76db0d64624bd43f0e538f0649d (DEFAULT_TABLE table=default.Table_884652a61536446f903e8a58dac9590f [id=5178ded07b844cf4abc1616c387b8585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.543345 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f1c76db0d64624bd43f0e538f0649d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.544770 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.545154 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_04f9f9491e5f49ec9660427758223e26" } modify_external_catalogs: true
I20250112 20:51:25.545325 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_04f9f9491e5f49ec9660427758223e26" } modify_external_catalogs: true
I20250112 20:51:25.545495 893 tablet_service.cc:1467] Processing CreateTablet for tablet e0f1c76db0d64624bd43f0e538f0649d (DEFAULT_TABLE table=default.Table_884652a61536446f903e8a58dac9590f [id=5178ded07b844cf4abc1616c387b8585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.545852 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f1c76db0d64624bd43f0e538f0649d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.545848 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ad0df4285db4bb6954f3e4c8d858115 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.546316 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3d5bec5466fb4f71bbe9b86ec73f874f" } modify_external_catalogs: true
I20250112 20:51:25.546468 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3d5bec5466fb4f71bbe9b86ec73f874f" } modify_external_catalogs: true
I20250112 20:51:25.546964 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 382aefb4a5e34999b560a9237839b380 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.547542 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ad0df4285db4bb6954f3e4c8d858115 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.548063 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2feab5fd4f5b4753ab157456f04f7e70" } modify_external_catalogs: true
I20250112 20:51:25.548202 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2feab5fd4f5b4753ab157456f04f7e70" } modify_external_catalogs: true
I20250112 20:51:25.548949 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f0c0c02b146a40f38d0098e259fe4cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.549193 710 leader_election.cc:343] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.548988 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.549448 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_42918b2644f14f05ae359f8a5c931c2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.549686 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:25.549470 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0c0c02b146a40f38d0098e259fe4cb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:25.549871 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:25.550361 710 leader_election.cc:343] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.550474 710 leader_election.cc:304] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:25.550868 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42918b2644f14f05ae359f8a5c931c2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.552086 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_18294fc7a4b448b2a1f57904a112b19d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.552232 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5dd7d40d429f4aa08fedf80d8acb42a7 (table default.Table_c9327af43e4f40429a836542ade34ab7 [id=fdf41f1792bb41e6bd764ee979d48a7f]) successfully deleted
W20250112 20:51:25.552543 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18294fc7a4b448b2a1f57904a112b19d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:25.552623 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 already in progress: creating tablet
I20250112 20:51:25.552922 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.553809 1169 raft_consensus.cc:2743] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.554559 1186 ts_tablet_manager.cc:1905] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.558037 1184 ts_tablet_manager.cc:1918] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.558153 1184 log.cc:1198] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/87756828be274286a3bb674040727828
I20250112 20:51:25.558485 1184 ts_tablet_manager.cc:1939] T 87756828be274286a3bb674040727828 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:25.560592 1185 ts_tablet_manager.cc:1918] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:25.560720 1185 log.cc:1198] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/87756828be274286a3bb674040727828
I20250112 20:51:25.561394 1185 ts_tablet_manager.cc:1939] T 87756828be274286a3bb674040727828 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:25.561606 1163 tablet_bootstrap.cc:492] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.561745 1163 ts_tablet_manager.cc:1397] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.302s user 0.091s sys 0.144s
I20250112 20:51:25.562587 1163 raft_consensus.cc:357] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.562800 1163 raft_consensus.cc:383] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.562919 1185 tablet_replica.cc:331] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:25.562909 1163 raft_consensus.cc:738] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.563158 1185 raft_consensus.cc:2238] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.563128 1163 consensus_queue.cc:260] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.563337 1185 raft_consensus.cc:2267] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.564494 1163 ts_tablet_manager.cc:1428] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:25.564750 1163 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.567854 1185 ts_tablet_manager.cc:1905] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:25.568177 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet aa3bd4eb92114bdb98b923fda9192cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa3bd4eb92114bdb98b923fda9192cd6 already in progress: creating tablet
I20250112 20:51:25.568658 1164 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.568886 1164 ts_tablet_manager.cc:1397] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.009s
I20250112 20:51:25.569731 1164 raft_consensus.cc:357] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.570024 1164 raft_consensus.cc:383] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.570161 1164 raft_consensus.cc:738] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.570427 1164 consensus_queue.cc:260] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.567727 1163 tablet_bootstrap.cc:654] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.571159 1164 ts_tablet_manager.cc:1428] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:25.571502 1164 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.573685 1164 tablet_bootstrap.cc:654] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.568733 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 87756828be274286a3bb674040727828 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
I20250112 20:51:25.581265 1163 tablet_bootstrap.cc:492] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.581436 1163 ts_tablet_manager.cc:1397] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250112 20:51:25.582217 1163 raft_consensus.cc:357] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.582446 1163 raft_consensus.cc:383] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.582520 1163 raft_consensus.cc:738] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.582767 1163 consensus_queue.cc:260] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.584328 1163 ts_tablet_manager.cc:1428] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:25.584663 1163 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:25.586614 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d616a8ad9ad4777b44deb77bc321d66 consensus state has no leader
I20250112 20:51:25.586670 1163 tablet_bootstrap.cc:654] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.598639 1164 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.598790 1164 ts_tablet_manager.cc:1397] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.007s
I20250112 20:51:25.599774 1164 raft_consensus.cc:357] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.600049 1164 raft_consensus.cc:383] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.600149 1164 raft_consensus.cc:738] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.600402 1164 consensus_queue.cc:260] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.601333 1164 ts_tablet_manager.cc:1428] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:25.601706 1164 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.603511 1169 raft_consensus.cc:491] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.603703 1169 raft_consensus.cc:513] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.604269 1169 leader_election.cc:290] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.604327 1164 tablet_bootstrap.cc:654] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.605046 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:25.605628 710 leader_election.cc:343] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.608115 1163 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.608472 1163 ts_tablet_manager.cc:1397] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20250112 20:51:25.609247 1163 raft_consensus.cc:357] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:25.613211 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e814b7a9b4cb4336b62b7cd7e35461b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e814b7a9b4cb4336b62b7cd7e35461b6 already in progress: creating tablet
I20250112 20:51:25.609443 1163 raft_consensus.cc:383] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.613440 1163 raft_consensus.cc:738] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.613683 1163 consensus_queue.cc:260] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.614284 1164 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.614457 1164 ts_tablet_manager.cc:1397] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 20:51:25.615090 1163 ts_tablet_manager.cc:1428] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:25.616724 1164 raft_consensus.cc:357] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.616972 1164 raft_consensus.cc:383] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.617050 1164 raft_consensus.cc:738] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.617286 1164 consensus_queue.cc:260] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.617658 1186 ts_tablet_manager.cc:1918] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.617558 1163 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.618652 1164 ts_tablet_manager.cc:1428] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:25.618992 1164 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.620535 1164 tablet_bootstrap.cc:654] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.620541 1163 tablet_bootstrap.cc:654] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.623045 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:25.623706 710 leader_election.cc:343] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.623971 1186 log.cc:1198] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/6f79679bd64041e69e8c15369ec4acfb
I20250112 20:51:25.624838 1178 tablet_bootstrap.cc:492] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.624990 1178 ts_tablet_manager.cc:1397] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20250112 20:51:25.625644 1178 raft_consensus.cc:357] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.625819 1178 raft_consensus.cc:383] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.625897 1178 raft_consensus.cc:738] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.626084 1178 consensus_queue.cc:260] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.626510 1178 ts_tablet_manager.cc:1428] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:25.626780 1178 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:25.613543 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 30e53e67076c4a5a89f1e93fc168a997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30e53e67076c4a5a89f1e93fc168a997 already in progress: creating tablet
W20250112 20:51:25.628371 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 382aefb4a5e34999b560a9237839b380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 382aefb4a5e34999b560a9237839b380 already in progress: creating tablet
W20250112 20:51:25.628532 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 616b8aa8fd62486c978c275712064f77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616b8aa8fd62486c978c275712064f77 already in progress: creating tablet
I20250112 20:51:25.629475 710 leader_election.cc:304] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.629492 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb (DEFAULT_TABLE table=default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.629875 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a692cc5fb6f4afc854c4821274edfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.630354 1169 raft_consensus.cc:2743] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.630668 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.631021 1186 ts_tablet_manager.cc:1939] T 6f79679bd64041e69e8c15369ec4acfb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:25.631898 1186 tablet_replica.cc:331] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.632050 1186 raft_consensus.cc:2238] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.632203 1186 raft_consensus.cc:2267] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.634016 1186 ts_tablet_manager.cc:1905] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.636842 1178 tablet_bootstrap.cc:654] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.638115 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb (DEFAULT_TABLE table=default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.638487 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a692cc5fb6f4afc854c4821274edfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.644040 1186 ts_tablet_manager.cc:1918] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.644641 1186 log.cc:1198] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e482a2543cdf4020a03f131c980e7928
I20250112 20:51:25.645133 1186 ts_tablet_manager.cc:1939] T e482a2543cdf4020a03f131c980e7928 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:25.646622 888 tablet_service.cc:1467] Processing CreateTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb (DEFAULT_TABLE table=default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.646992 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a692cc5fb6f4afc854c4821274edfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.653995 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.654294 753 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.654402 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.654660 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.666643 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d45f925e122a4a0d87d36700dddbdf92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.667227 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d83a3643f7974d1ab7d542bdc1dea5ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.667917 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d83a3643f7974d1ab7d542bdc1dea5ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:25.669723 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45f925e122a4a0d87d36700dddbdf92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.681377 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ea7336d13ea46748167e35e7e7e8c98 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.682153 1178 tablet_bootstrap.cc:492] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.682745 1178 ts_tablet_manager.cc:1397] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.008s
I20250112 20:51:25.683609 1178 raft_consensus.cc:357] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.683874 1178 raft_consensus.cc:383] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.683941 1178 raft_consensus.cc:738] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.684171 1178 consensus_queue.cc:260] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.689154 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:25.689697 450 catalog_manager.cc:6596] ReplaceTablet: tablet 8ad0df4285db4bb6954f3e4c8d858115 deleted and replaced by tablet 7f39195945024f4fba9942faec70bc31
I20250112 20:51:25.692971 1178 ts_tablet_manager.cc:1428] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 20:51:25.693343 1185 ts_tablet_manager.cc:1918] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.693511 1185 log.cc:1198] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e814b7a9b4cb4336b62b7cd7e35461b6
I20250112 20:51:25.693924 1185 ts_tablet_manager.cc:1939] T e814b7a9b4cb4336b62b7cd7e35461b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:25.698453 1178 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.700299 1178 tablet_bootstrap.cc:654] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.703680 1185 tablet_replica.cc:331] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:25.703930 1185 raft_consensus.cc:2238] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.704139 1185 raft_consensus.cc:2267] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.707244 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.707747 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.713654 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac (DEFAULT_TABLE table=default.Table_598b0b1981fc4153baa6e9b8b2521981 [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.713743 1185 ts_tablet_manager.cc:1905] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.713984 756 tablet_service.cc:1467] Processing CreateTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac (DEFAULT_TABLE table=default.Table_598b0b1981fc4153baa6e9b8b2521981 [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.714112 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b70ffb3e3341d398d36ff76993e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.714373 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b70ffb3e3341d398d36ff76993e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.720567 1178 tablet_bootstrap.cc:492] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.720731 1178 ts_tablet_manager.cc:1397] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250112 20:51:25.721598 1178 raft_consensus.cc:357] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.721834 1178 raft_consensus.cc:383] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.721936 1178 raft_consensus.cc:738] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.726393 893 tablet_service.cc:1467] Processing CreateTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac (DEFAULT_TABLE table=default.Table_598b0b1981fc4153baa6e9b8b2521981 [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.726764 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b70ffb3e3341d398d36ff76993e9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.727083 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b (DEFAULT_TABLE table=default.Table_dca6dbc93bff4fccaf0c975522fa3f74 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.727533 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afa5b0717df4f83a2b44efac12fc06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.729086 1178 consensus_queue.cc:260] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.733018 1178 ts_tablet_manager.cc:1428] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:25.743530 1178 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.746198 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b (DEFAULT_TABLE table=default.Table_dca6dbc93bff4fccaf0c975522fa3f74 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.746750 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afa5b0717df4f83a2b44efac12fc06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.747889 1178 tablet_bootstrap.cc:654] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.750871 1185 ts_tablet_manager.cc:1918] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.751006 1185 log.cc:1198] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/30e53e67076c4a5a89f1e93fc168a997
I20250112 20:51:25.751426 1185 ts_tablet_manager.cc:1939] T 30e53e67076c4a5a89f1e93fc168a997 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:25.751693 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (DEFAULT_TABLE table=default.Table_71341c6c973c4559949e989d69912a93 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.751972 752 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (DEFAULT_TABLE table=default.Table_71341c6c973c4559949e989d69912a93 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.752123 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa5246382745ecbea4b9bd4825e8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.752874 754 tablet_service.cc:1467] Processing CreateTablet for tablet b13d3f3e597f4876bce5852961f37ea0 (DEFAULT_TABLE table=default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.753620 893 tablet_service.cc:1467] Processing CreateTablet for tablet b13d3f3e597f4876bce5852961f37ea0 (DEFAULT_TABLE table=default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.753985 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d3f3e597f4876bce5852961f37ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.757138 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b13d3f3e597f4876bce5852961f37ea0 (DEFAULT_TABLE table=default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.757498 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d3f3e597f4876bce5852961f37ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.758122 1178 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.758298 1178 ts_tablet_manager.cc:1397] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250112 20:51:25.758769 1169 raft_consensus.cc:491] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.758919 1169 raft_consensus.cc:513] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.759405 1178 raft_consensus.cc:357] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.759546 1169 leader_election.cc:290] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.759641 1178 raft_consensus.cc:383] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.759707 1178 raft_consensus.cc:738] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:25.760489 710 leader_election.cc:343] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 6f79679bd64041e69e8c15369ec4acfb
I20250112 20:51:25.761240 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79679bd64041e69e8c15369ec4acfb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:25.761437 1049 raft_consensus.cc:2463] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:25.761476 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa5246382745ecbea4b9bd4825e8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.761847 710 leader_election.cc:304] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.762296 1169 raft_consensus.cc:2798] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:25.762408 1169 raft_consensus.cc:491] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:25.762524 1169 raft_consensus.cc:3054] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.763095 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d3f3e597f4876bce5852961f37ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.767773 1169 raft_consensus.cc:513] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:25.769752 710 leader_election.cc:343] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 6f79679bd64041e69e8c15369ec4acfb
W20250112 20:51:25.770129 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0871d4272e874a958174f96bfc0e1a95 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0871d4272e874a958174f96bfc0e1a95 consensus state has no leader
I20250112 20:51:25.770915 1169 leader_election.cc:290] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.770967 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f79679bd64041e69e8c15369ec4acfb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:25.771158 1049 raft_consensus.cc:3054] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.771373 891 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (DEFAULT_TABLE table=default.Table_71341c6c973c4559949e989d69912a93 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.771690 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa5246382745ecbea4b9bd4825e8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.773923 1049 raft_consensus.cc:2463] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:25.774358 710 leader_election.cc:304] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.775043 1169 raft_consensus.cc:2798] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:25.775622 1169 raft_consensus.cc:695] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:25.776337 1169 consensus_queue.cc:237] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.778510 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b (DEFAULT_TABLE table=default.Table_dca6dbc93bff4fccaf0c975522fa3f74 [id=61a7256a26de426ca7f8e0763013ac26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.779044 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afa5b0717df4f83a2b44efac12fc06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.779914 1178 consensus_queue.cc:260] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.783434 1178 ts_tablet_manager.cc:1428] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250112 20:51:25.790864 893 tablet_service.cc:1467] Processing CreateTablet for tablet decdec0d34344ec4b422643b0d4e7659 (DEFAULT_TABLE table=default.Table_b0367c09c21647a4beea62b7dd6d3d19 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.791167 1178 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.791313 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decdec0d34344ec4b422643b0d4e7659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.793573 1178 tablet_bootstrap.cc:654] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.796705 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:25.797667 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d48b17628cf145b69e5fdde4131d9fd2
I20250112 20:51:25.799423 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:25.803004 1178 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.803160 1178 ts_tablet_manager.cc:1397] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250112 20:51:25.803257 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:25.804032 1178 raft_consensus.cc:357] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.804277 1178 raft_consensus.cc:383] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.805719 1021 tablet_service.cc:1467] Processing CreateTablet for tablet decdec0d34344ec4b422643b0d4e7659 (DEFAULT_TABLE table=default.Table_b0367c09c21647a4beea62b7dd6d3d19 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.806078 754 tablet_service.cc:1467] Processing CreateTablet for tablet decdec0d34344ec4b422643b0d4e7659 (DEFAULT_TABLE table=default.Table_b0367c09c21647a4beea62b7dd6d3d19 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.804378 1178 raft_consensus.cc:738] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.806566 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decdec0d34344ec4b422643b0d4e7659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.806672 1178 consensus_queue.cc:260] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.811223 1178 ts_tablet_manager.cc:1428] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20250112 20:51:25.814457 1178 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.816905 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:25.820294 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:25.822619 1178 tablet_bootstrap.cc:654] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.823086 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8b414be57990432bb786daf5faf0cc2e (DEFAULT_TABLE table=default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.823508 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b414be57990432bb786daf5faf0cc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.824649 754 tablet_service.cc:1467] Processing CreateTablet for tablet 8b414be57990432bb786daf5faf0cc2e (DEFAULT_TABLE table=default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.825134 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b414be57990432bb786daf5faf0cc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.829612 887 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 (DEFAULT_TABLE table=default.Table_2a522da7a096468db3231e0f9f8205bc [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.830029 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca061b82a894e91b9508613ccd8ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.833832 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 (DEFAULT_TABLE table=default.Table_2a522da7a096468db3231e0f9f8205bc [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.834206 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca061b82a894e91b9508613ccd8ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.835729 886 tablet_service.cc:1467] Processing CreateTablet for tablet cc971026f6b44db6af8215890ba71075 (DEFAULT_TABLE table=default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.836884 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc971026f6b44db6af8215890ba71075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.845013 1227 raft_consensus.cc:491] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.845206 1227 raft_consensus.cc:513] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.846149 1227 leader_election.cc:290] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:25.847188 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:25.847952 710 leader_election.cc:343] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.851233 1178 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.852298 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 87756828be274286a3bb674040727828 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
I20250112 20:51:25.852941 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e814b7a9b4cb4336b62b7cd7e35461b6 (table default.Table_5bfc59824f23459cade39fdb87c63491 [id=8ffaa9b9ac214c9f9d80dcc8865287a8]) successfully deleted
I20250112 20:51:25.853647 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 30e53e67076c4a5a89f1e93fc168a997 (table default.Table_dca6dbc93bff4fccaf0c975522fa3f74 [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
W20250112 20:51:25.853981 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 382aefb4a5e34999b560a9237839b380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 382aefb4a5e34999b560a9237839b380 already in progress: creating tablet
W20250112 20:51:25.854648 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 616b8aa8fd62486c978c275712064f77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616b8aa8fd62486c978c275712064f77 already in progress: creating tablet
I20250112 20:51:25.855150 752 tablet_service.cc:1467] Processing CreateTablet for tablet cc971026f6b44db6af8215890ba71075 (DEFAULT_TABLE table=default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:25.855660 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 616b8aa8fd62486c978c275712064f77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 616b8aa8fd62486c978c275712064f77 already in progress: creating tablet
I20250112 20:51:25.855859 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc971026f6b44db6af8215890ba71075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.856024 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 6f79679bd64041e69e8c15369ec4acfb (table default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]) successfully deleted
I20250112 20:51:25.856230 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e482a2543cdf4020a03f131c980e7928 (table default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]) successfully deleted
I20250112 20:51:25.861498 1178 ts_tablet_manager.cc:1397] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20250112 20:51:25.861928 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8b414be57990432bb786daf5faf0cc2e (DEFAULT_TABLE table=default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.862146 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e7a8468369af418ea4a1b0a681f8ff74
I20250112 20:51:25.862344 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decdec0d34344ec4b422643b0d4e7659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.863337 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b414be57990432bb786daf5faf0cc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.865129 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 (DEFAULT_TABLE table=default.Table_2a522da7a096468db3231e0f9f8205bc [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.865490 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cc971026f6b44db6af8215890ba71075 (DEFAULT_TABLE table=default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.865041 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:25.866060 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca061b82a894e91b9508613ccd8ecd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.863310 1178 raft_consensus.cc:357] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.866263 1178 raft_consensus.cc:383] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.866359 1178 raft_consensus.cc:738] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.866645 1178 consensus_queue.cc:260] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.869093 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc971026f6b44db6af8215890ba71075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.872494 1178 ts_tablet_manager.cc:1428] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:25.885360 1178 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.888005 1178 tablet_bootstrap.cc:654] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.889539 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:25.889972 447 catalog_manager.cc:6596] ReplaceTablet: tablet 5ea7336d13ea46748167e35e7e7e8c98 deleted and replaced by tablet 33a90f801d854d4d8f7f09112e2d2929
I20250112 20:51:25.895102 1163 tablet_bootstrap.cc:492] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.895272 1163 ts_tablet_manager.cc:1397] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.280s user 0.046s sys 0.102s
I20250112 20:51:25.896020 1163 raft_consensus.cc:357] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:25.896237 1163 raft_consensus.cc:383] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.896306 1163 raft_consensus.cc:738] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.896555 1163 consensus_queue.cc:260] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:25.897321 710 leader_election.cc:343] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.897486 710 leader_election.cc:304] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.898116 1227 raft_consensus.cc:2743] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.898420 1169 raft_consensus.cc:491] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.898561 1169 raft_consensus.cc:513] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.899209 1169 leader_election.cc:290] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:25.899895 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:25.900072 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:25.901239 1163 ts_tablet_manager.cc:1428] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250112 20:51:25.901360 710 leader_election.cc:343] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.901460 1178 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.901592 1178 ts_tablet_manager.cc:1397] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
W20250112 20:51:25.901584 710 leader_election.cc:343] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.902036 710 leader_election.cc:304] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.902452 1169 raft_consensus.cc:2743] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.903434 1164 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.903574 1164 ts_tablet_manager.cc:1397] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.285s user 0.013s sys 0.034s
W20250112 20:51:25.904722 1218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:25.905162 1218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250112 20:51:25.905043 1164 raft_consensus.cc:357] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.905226 1164 raft_consensus.cc:383] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:25.905295 1218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 20:51:25.905303 1164 raft_consensus.cc:738] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.905493 1164 consensus_queue.cc:260] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.908061 1163 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.909250 1164 ts_tablet_manager.cc:1428] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250112 20:51:25.911432 1163 tablet_bootstrap.cc:654] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.911747 1164 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.914528 1164 tablet_bootstrap.cc:654] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.915457 1178 raft_consensus.cc:357] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.915669 1178 raft_consensus.cc:383] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.915760 1178 raft_consensus.cc:738] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.915998 1178 consensus_queue.cc:260] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:25.918285 1218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:25.918432 1218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:25.918517 1218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:25.918601 1218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:25.919445 1178 ts_tablet_manager.cc:1428] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 20:51:25.921836 1163 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.922044 1163 ts_tablet_manager.cc:1397] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20250112 20:51:25.922461 1178 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.922923 1163 raft_consensus.cc:357] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.923208 1163 raft_consensus.cc:383] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.923341 1163 raft_consensus.cc:738] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.923632 1163 consensus_queue.cc:260] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.924084 1178 tablet_bootstrap.cc:654] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.926704 1163 ts_tablet_manager.cc:1428] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250112 20:51:25.928421 1164 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.928596 1164 ts_tablet_manager.cc:1397] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250112 20:51:25.928822 1228 raft_consensus.cc:491] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.928951 1228 raft_consensus.cc:513] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.929059 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1d08a9783214c518f1eda3d907286a1" } modify_external_catalogs: true
I20250112 20:51:25.929244 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1d08a9783214c518f1eda3d907286a1" } modify_external_catalogs: true
I20250112 20:51:25.929661 1228 leader_election.cc:290] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:25.930042 1163 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.930223 447 master_service.cc:946] ReplaceTablet: received request to replace tablet aa3bd4eb92114bdb98b923fda9192cd6 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:25.930749 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:25.930943 776 raft_consensus.cc:2463] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:25.931109 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:25.931764 847 leader_election.cc:304] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:25.932055 847 leader_election.cc:343] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.932097 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1b2cd38a56934793bb10f3381448a3ff" } modify_external_catalogs: true
I20250112 20:51:25.932174 1163 tablet_bootstrap.cc:654] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.932269 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1b2cd38a56934793bb10f3381448a3ff" } modify_external_catalogs: true
I20250112 20:51:25.932329 1228 raft_consensus.cc:2798] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:25.932425 1228 raft_consensus.cc:491] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:25.932494 1228 raft_consensus.cc:3054] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.933226 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_79cfd73169d6487388af1340e2b19a9e" } new_table_name: "default.Table_9cf0195d130e459fbc8e651f57546be7" modify_external_catalogs: true
I20250112 20:51:25.934928 1169 raft_consensus.cc:491] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:25.935067 1169 raft_consensus.cc:513] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.935636 1169 leader_election.cc:290] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:25.936082 1218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=39277
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:25.936470 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:25.937134 710 leader_election.cc:343] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.937443 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:25.937958 710 leader_election.cc:343] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:25.938069 710 leader_election.cc:304] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:25.938174 1218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:25.938467 1169 raft_consensus.cc:2743] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:25.939214 1228 raft_consensus.cc:513] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.940354 1218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:51:25.940310 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:25.940868 847 leader_election.cc:343] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:25.941510 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc9aa4f42f89455cba7c72e7a49c6a94 consensus state has no leader
I20250112 20:51:25.941608 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:25.941799 776 raft_consensus.cc:3054] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:25.944361 776 raft_consensus.cc:2463] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:25.944924 847 leader_election.cc:304] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:25.945210 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dca6dbc93bff4fccaf0c975522fa3f74" } new_table_name: "default.Table_ca9e0810a7294c4fab5de64b7fde63af" modify_external_catalogs: true
I20250112 20:51:25.945658 1228 leader_election.cc:290] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:25.945598 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ceaaef5837a2461f90a9ca600452abf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.945851 1228 raft_consensus.cc:2798] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:25.945979 1228 raft_consensus.cc:695] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
W20250112 20:51:25.946126 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceaaef5837a2461f90a9ca600452abf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.946341 1228 consensus_queue.cc:237] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.947237 1163 tablet_bootstrap.cc:492] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.947431 1163 ts_tablet_manager.cc:1397] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20250112 20:51:25.948230 1163 raft_consensus.cc:357] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.948629 1163 raft_consensus.cc:383] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.948760 1163 raft_consensus.cc:738] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.949134 1163 consensus_queue.cc:260] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.951342 1164 raft_consensus.cc:357] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.951637 1164 raft_consensus.cc:383] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.951740 1164 raft_consensus.cc:738] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:25.951965 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2afa5b0717df4f83a2b44efac12fc06b consensus state has no leader
I20250112 20:51:25.952556 1163 ts_tablet_manager.cc:1428] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:25.952761 1164 consensus_queue.cc:260] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.955597 1164 ts_tablet_manager.cc:1428] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.027s user 0.000s sys 0.003s
I20250112 20:51:25.955641 1163 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.955989 1164 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.957692 1163 tablet_bootstrap.cc:654] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.962163 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b0367c09c21647a4beea62b7dd6d3d19" } new_table_name: "default.Table_f1e1ad1b8b6a487ca4a40ae50884b72d" modify_external_catalogs: true
I20250112 20:51:25.964082 1178 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:25.963950 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_22423ed5e6774aa6b41f72eb92b3ab3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:25.964262 1178 ts_tablet_manager.cc:1397] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.003s
W20250112 20:51:25.964617 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22423ed5e6774aa6b41f72eb92b3ab3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.965094 1178 raft_consensus.cc:357] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.965389 1178 raft_consensus.cc:383] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.965480 1178 raft_consensus.cc:738] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.966250 1164 tablet_bootstrap.cc:654] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.966318 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9d07ceff6ea140e997aff9afbaf71e91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.966902 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d07ceff6ea140e997aff9afbaf71e91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.968437 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fb52d5971de44b71b7a53e1d05798b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:25.968961 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb52d5971de44b71b7a53e1d05798b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:25.972288 1163 tablet_bootstrap.cc:492] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:25.972218 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6c0fcd020f404207804bc1fd94676779" } new_table_name: "default.Table_20471004d83e485dbea6520c7bdb37b0" modify_external_catalogs: true
I20250112 20:51:25.972162 1178 consensus_queue.cc:260] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.972440 1163 ts_tablet_manager.cc:1397] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20250112 20:51:25.973304 1178 ts_tablet_manager.cc:1428] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 20:51:25.973289 1163 raft_consensus.cc:357] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.973495 1163 raft_consensus.cc:383] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.973605 1178 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:25.973594 1163 raft_consensus.cc:738] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.973804 1163 consensus_queue.cc:260] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:25.974386 1163 ts_tablet_manager.cc:1428] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250112 20:51:25.974858 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:25.975744 1163 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:25.975842 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.976231 1178 tablet_bootstrap.cc:654] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.976392 1184 tablet_replica.cc:331] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:25.976596 1184 raft_consensus.cc:2238] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.976809 1184 raft_consensus.cc:2267] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.977281 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.977375 1184 ts_tablet_manager.cc:1905] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.977861 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.977957 1164 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:25.978134 1164 ts_tablet_manager.cc:1397] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250112 20:51:25.979454 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.979516 1163 tablet_bootstrap.cc:654] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:25.980041 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.980456 1164 raft_consensus.cc:357] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.980716 1164 raft_consensus.cc:383] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:25.980829 1164 raft_consensus.cc:738] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:25.981251 1164 consensus_queue.cc:260] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:25.983275 1164 ts_tablet_manager.cc:1428] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:51:25.983664 1164 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:25.985702 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 87756828be274286a3bb674040727828 with delete_type TABLET_DATA_DELETED (replaced by tablet 0871d4272e874a958174f96bfc0e1a95) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.985945 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.986184 1186 tablet_replica.cc:331] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.986400 1186 raft_consensus.cc:2238] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.986574 1186 raft_consensus.cc:2267] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:25.987773 1186 ts_tablet_manager.cc:1905] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:25.988080 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.988870 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.989424 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.990130 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.990923 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.991506 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.992282 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.993212 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:25.993791 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:25.994743 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 45484f8e48524b86a6f418b88a71ac18 (DEFAULT_TABLE table=default.Table_18294fc7a4b448b2a1f57904a112b19d [id=eba58b5d38e8443e842cca705dc51aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:25.995126 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45484f8e48524b86a6f418b88a71ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:25.996500 1186 ts_tablet_manager.cc:1918] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:25.997287 1186 log.cc:1198] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/87756828be274286a3bb674040727828
I20250112 20:51:25.998306 1186 ts_tablet_manager.cc:1939] T 87756828be274286a3bb674040727828 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:25.999531 1186 tablet_replica.cc:331] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:25.999786 1186 raft_consensus.cc:2238] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:25.999992 1186 raft_consensus.cc:2267] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.000715 1186 ts_tablet_manager.cc:1905] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.004616 1164 tablet_bootstrap.cc:654] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.005471 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.005919 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.006624 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.011758 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:26.011849 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet decdec0d34344ec4b422643b0d4e7659 consensus state has no leader
I20250112 20:51:26.013278 1184 ts_tablet_manager.cc:1918] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.013762 1184 log.cc:1198] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:26.014190 1184 ts_tablet_manager.cc:1939] T df9fcf5bc7584cdcb17b19fc69802b84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.015268 893 tablet_service.cc:1514] Processing DeleteTablet for tablet df9fcf5bc7584cdcb17b19fc69802b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.015584 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.015854 1185 tablet_replica.cc:331] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.016058 1185 raft_consensus.cc:2238] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.016216 1185 raft_consensus.cc:2267] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.016620 1184 tablet_replica.cc:331] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.016863 1184 raft_consensus.cc:2238] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.016990 1185 ts_tablet_manager.cc:1905] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.017144 1184 raft_consensus.cc:2267] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.017679 1184 ts_tablet_manager.cc:1905] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.023257 1185 ts_tablet_manager.cc:1918] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.023408 1185 log.cc:1198] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:26.023801 1185 ts_tablet_manager.cc:1939] T 8ad0df4285db4bb6954f3e4c8d858115 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.024830 1163 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.025117 1163 ts_tablet_manager.cc:1397] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.009s
I20250112 20:51:26.025698 1164 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.025883 1164 ts_tablet_manager.cc:1397] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.000s
I20250112 20:51:26.026139 1163 raft_consensus.cc:357] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.026528 1163 raft_consensus.cc:383] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.026661 1163 raft_consensus.cc:738] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.026770 1164 raft_consensus.cc:357] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.027035 1164 raft_consensus.cc:383] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.027122 1164 raft_consensus.cc:738] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.026960 1163 consensus_queue.cc:260] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.028260 1178 tablet_bootstrap.cc:492] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.028434 1178 ts_tablet_manager.cc:1397] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.006s
I20250112 20:51:26.028731 1163 ts_tablet_manager.cc:1428] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:26.029093 1163 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.029464 1178 raft_consensus.cc:357] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.029717 1178 raft_consensus.cc:383] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.029816 1178 raft_consensus.cc:738] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.030045 1178 consensus_queue.cc:260] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.031818 1178 ts_tablet_manager.cc:1428] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:26.028188 1184 ts_tablet_manager.cc:1918] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.033757 1164 consensus_queue.cc:260] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.034238 1184 log.cc:1198] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/6f79679bd64041e69e8c15369ec4acfb
I20250112 20:51:26.034663 1184 ts_tablet_manager.cc:1939] T 6f79679bd64041e69e8c15369ec4acfb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.038381 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7f39195945024f4fba9942faec70bc31 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.038558 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9c55b922c39545ac9c89115cd587ea8c (DEFAULT_TABLE table=default.Table_42918b2644f14f05ae359f8a5c931c2a [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.038765 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39195945024f4fba9942faec70bc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.038933 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c55b922c39545ac9c89115cd587ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.040525 1178 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.041065 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8c697bba271f4455956b9a2511155b79 (DEFAULT_TABLE table=default.Table_f0c0c02b146a40f38d0098e259fe4cb6 [id=fc1d61ae0aec4a519360241bfcae3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.041412 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c697bba271f4455956b9a2511155b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.041339 893 tablet_service.cc:1467] Processing CreateTablet for tablet 45484f8e48524b86a6f418b88a71ac18 (DEFAULT_TABLE table=default.Table_18294fc7a4b448b2a1f57904a112b19d [id=eba58b5d38e8443e842cca705dc51aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.041728 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45484f8e48524b86a6f418b88a71ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.041066 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7f39195945024f4fba9942faec70bc31 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.042688 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39195945024f4fba9942faec70bc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.042313 1164 ts_tablet_manager.cc:1428] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250112 20:51:26.043228 1164 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.043613 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8c697bba271f4455956b9a2511155b79 (DEFAULT_TABLE table=default.Table_f0c0c02b146a40f38d0098e259fe4cb6 [id=fc1d61ae0aec4a519360241bfcae3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.043970 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c697bba271f4455956b9a2511155b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.044857 888 tablet_service.cc:1467] Processing CreateTablet for tablet 7f39195945024f4fba9942faec70bc31 (DEFAULT_TABLE table=default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.045208 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39195945024f4fba9942faec70bc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.049023 754 tablet_service.cc:1467] Processing CreateTablet for tablet 45484f8e48524b86a6f418b88a71ac18 (DEFAULT_TABLE table=default.Table_18294fc7a4b448b2a1f57904a112b19d [id=eba58b5d38e8443e842cca705dc51aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.051831 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 9c55b922c39545ac9c89115cd587ea8c (DEFAULT_TABLE table=default.Table_42918b2644f14f05ae359f8a5c931c2a [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.051904 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ef89229192934339954566e366b6891c (DEFAULT_TABLE table=default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.052232 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef89229192934339954566e366b6891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.052898 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45484f8e48524b86a6f418b88a71ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.053769 756 tablet_service.cc:1467] Processing CreateTablet for tablet ef89229192934339954566e366b6891c (DEFAULT_TABLE table=default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.055876 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef89229192934339954566e366b6891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.058492 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c55b922c39545ac9c89115cd587ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:26.059787 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb1dffbe43ba4eeab7ee12d901b62ffb consensus state has no leader
I20250112 20:51:26.061728 1163 tablet_bootstrap.cc:654] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:26.063131 1274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:26.063158 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9c55b922c39545ac9c89115cd587ea8c (DEFAULT_TABLE table=default.Table_42918b2644f14f05ae359f8a5c931c2a [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.063575 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c55b922c39545ac9c89115cd587ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.063891 892 tablet_service.cc:1467] Processing CreateTablet for tablet ef89229192934339954566e366b6891c (DEFAULT_TABLE table=default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.067382 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef89229192934339954566e366b6891c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.068249 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet df9fcf5bc7584cdcb17b19fc69802b84 (table default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1 [id=a578996c09484fb1a6b08eb55eff74d0]) successfully deleted
I20250112 20:51:26.068439 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 6f79679bd64041e69e8c15369ec4acfb (table default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]) successfully deleted
W20250112 20:51:26.068661 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5dd7d40d429f4aa08fedf80d8acb42a7 because the tablet was not found. No further retry: Not found: Tablet not found: 5dd7d40d429f4aa08fedf80d8acb42a7
W20250112 20:51:26.068913 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8ad0df4285db4bb6954f3e4c8d858115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad0df4285db4bb6954f3e4c8d858115 already in progress: creating tablet
W20250112 20:51:26.069157 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b13d3f3e597f4876bce5852961f37ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b13d3f3e597f4876bce5852961f37ea0 already in progress: creating tablet
W20250112 20:51:26.069409 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d48b17628cf145b69e5fdde4131d9fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d48b17628cf145b69e5fdde4131d9fd2 already in progress: creating tablet
W20250112 20:51:26.069648 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e7a8468369af418ea4a1b0a681f8ff74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7a8468369af418ea4a1b0a681f8ff74 already in progress: creating tablet
W20250112 20:51:26.069851 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5ea7336d13ea46748167e35e7e7e8c98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ea7336d13ea46748167e35e7e7e8c98 already in progress: creating tablet
W20250112 20:51:26.070523 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b13d3f3e597f4876bce5852961f37ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b13d3f3e597f4876bce5852961f37ea0 already in progress: creating tablet
W20250112 20:51:26.070744 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5ea7336d13ea46748167e35e7e7e8c98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ea7336d13ea46748167e35e7e7e8c98 already in progress: creating tablet
W20250112 20:51:26.070976 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e7a8468369af418ea4a1b0a681f8ff74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7a8468369af418ea4a1b0a681f8ff74 already in progress: creating tablet
W20250112 20:51:26.071161 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d48b17628cf145b69e5fdde4131d9fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d48b17628cf145b69e5fdde4131d9fd2 already in progress: creating tablet
I20250112 20:51:26.071359 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8ad0df4285db4bb6954f3e4c8d858115 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
W20250112 20:51:26.071786 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8ad0df4285db4bb6954f3e4c8d858115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad0df4285db4bb6954f3e4c8d858115 already in progress: creating tablet
I20250112 20:51:26.071990 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 87756828be274286a3bb674040727828 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
W20250112 20:51:26.072221 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b13d3f3e597f4876bce5852961f37ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b13d3f3e597f4876bce5852961f37ea0 already in progress: creating tablet
W20250112 20:51:26.072501 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d48b17628cf145b69e5fdde4131d9fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d48b17628cf145b69e5fdde4131d9fd2 already in progress: creating tablet
I20250112 20:51:26.078665 1164 tablet_bootstrap.cc:654] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.078958 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c2a8f71233d04ca39e49932308e56492 (DEFAULT_TABLE table=default.Table_d45f925e122a4a0d87d36700dddbdf92 [id=5e31685a4caa41729b85ec6b230333f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.079373 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a8f71233d04ca39e49932308e56492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.080924 1218 server_base.cc:1034] running on GCE node
I20250112 20:51:26.082593 751 tablet_service.cc:1467] Processing CreateTablet for tablet c2a8f71233d04ca39e49932308e56492 (DEFAULT_TABLE table=default.Table_d45f925e122a4a0d87d36700dddbdf92 [id=5e31685a4caa41729b85ec6b230333f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.082934 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.083282 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32dc02497f64a2e9ad99da6c9115d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.084375 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 313a5c9905bc445893f1ffa8752ca589 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.084470 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_65e847e586ae4fc99dee908b16e66be2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:26.085139 750 tablet_service.cc:1467] Processing CreateTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 (DEFAULT_TABLE table=default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.085314 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a8f71233d04ca39e49932308e56492. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:26.085711 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65e847e586ae4fc99dee908b16e66be2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.085844 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_29dce4195b10489a9cc62c236b93b8f7" } new_table_name: "default.Table_afbd47897b344f1db9916ab7e703ab36" modify_external_catalogs: true
I20250112 20:51:26.085918 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 (DEFAULT_TABLE table=default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.086256 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a90f801d854d4d8f7f09112e2d2929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.088322 749 tablet_service.cc:1467] Processing CreateTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.088702 1163 tablet_bootstrap.cc:492] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.088891 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a90f801d854d4d8f7f09112e2d2929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.088882 1163 ts_tablet_manager.cc:1397] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20250112 20:51:26.089653 1163 raft_consensus.cc:357] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.089789 1164 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.089850 1163 raft_consensus.cc:383] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.089946 1163 raft_consensus.cc:738] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.090256 1164 ts_tablet_manager.cc:1397] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.006s
I20250112 20:51:26.090179 1163 consensus_queue.cc:260] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.090262 1169 raft_consensus.cc:491] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.090655 1163 ts_tablet_manager.cc:1428] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.090904 1163 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.090924 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32dc02497f64a2e9ad99da6c9115d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.091243 1169 raft_consensus.cc:513] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.093580 1163 tablet_bootstrap.cc:654] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.094537 1169 leader_election.cc:290] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.095288 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e040383f6e2a4ccf9614fd2d119a3cb4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.095588 1049 raft_consensus.cc:2463] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:26.096167 710 leader_election.cc:304] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.096469 1169 raft_consensus.cc:2798] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.096575 1169 raft_consensus.cc:491] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.096664 1169 raft_consensus.cc:3054] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.100152 1186 ts_tablet_manager.cc:1918] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.100296 1186 log.cc:1198] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e814b7a9b4cb4336b62b7cd7e35461b6
W20250112 20:51:26.100412 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:26.101445 1164 raft_consensus.cc:357] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.101658 1164 raft_consensus.cc:383] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.101970 1218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:26.102705 1164 raft_consensus.cc:738] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.102147 1228 raft_consensus.cc:491] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.104215 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.104336 1218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:26.105106 1163 tablet_bootstrap.cc:492] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.105263 1163 ts_tablet_manager.cc:1397] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20250112 20:51:26.105569 1218 hybrid_clock.cc:648] HybridClock initialized: now 1736715086105487 us; error 58 us; skew 500 ppm
I20250112 20:51:26.105931 1218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:26.105516 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e040383f6e2a4ccf9614fd2d119a3cb4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.106678 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.108002 1228 raft_consensus.cc:513] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.108258 1163 raft_consensus.cc:357] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.108462 1163 raft_consensus.cc:383] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.108503 1186 ts_tablet_manager.cc:1939] T e814b7a9b4cb4336b62b7cd7e35461b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.108537 1163 raft_consensus.cc:738] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.108837 1163 consensus_queue.cc:260] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.109395 1163 ts_tablet_manager.cc:1428] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:26.111240 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8c697bba271f4455956b9a2511155b79 (DEFAULT_TABLE table=default.Table_f0c0c02b146a40f38d0098e259fe4cb6 [id=fc1d61ae0aec4a519360241bfcae3281]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.111717 1163 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.112010 1164 consensus_queue.cc:260] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:26.114805 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250112 20:51:26.115160 911 raft_consensus.cc:2463] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:26.116096 1164 ts_tablet_manager.cc:1428] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250112 20:51:26.116403 1164 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.116833 1163 tablet_bootstrap.cc:654] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.117367 890 tablet_service.cc:1467] Processing CreateTablet for tablet c2a8f71233d04ca39e49932308e56492 (DEFAULT_TABLE table=default.Table_d45f925e122a4a0d87d36700dddbdf92 [id=5e31685a4caa41729b85ec6b230333f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.118412 1218 webserver.cc:458] Webserver started at http://127.0.104.61:39277/ using document root <none> and password file <none>
I20250112 20:51:26.119354 886 tablet_service.cc:1467] Processing CreateTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.119438 1218 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:26.119582 1218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:26.120872 1163 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.121011 1163 ts_tablet_manager.cc:1397] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20250112 20:51:26.121752 1163 raft_consensus.cc:357] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.121946 1163 raft_consensus.cc:383] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.122017 1163 raft_consensus.cc:738] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.122216 1163 consensus_queue.cc:260] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:26.122318 847 leader_election.cc:343] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:26.122716 1228 leader_election.cc:290] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.122689 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e814b7a9b4cb4336b62b7cd7e35461b6 (table default.Table_5bfc59824f23459cade39fdb87c63491 [id=8ffaa9b9ac214c9f9d80dcc8865287a8]) successfully deleted
I20250112 20:51:26.122783 1163 ts_tablet_manager.cc:1428] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.123178 884 tablet_service.cc:1467] Processing CreateTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 (DEFAULT_TABLE table=default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.123313 1163 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:26.123536 847 leader_election.cc:343] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:26.125046 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c697bba271f4455956b9a2511155b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.125222 1163 tablet_bootstrap.cc:654] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.126082 1186 tablet_replica.cc:331] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.123677 847 leader_election.cc:304] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.127452 1186 raft_consensus.cc:2238] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.127995 1186 pending_rounds.cc:62] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:26.128224 1186 pending_rounds.cc:69] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584992048046080 op_type: NO_OP noop_request { }
I20250112 20:51:26.128420 1186 raft_consensus.cc:2883] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.128579 1186 raft_consensus.cc:2267] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:26.130040 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e7a8468369af418ea4a1b0a681f8ff74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7a8468369af418ea4a1b0a681f8ff74 already in progress: creating tablet
I20250112 20:51:26.132864 1228 raft_consensus.cc:491] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.133039 1228 raft_consensus.cc:513] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.134752 1228 leader_election.cc:290] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.134759 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c31c29073aa46ffa847905be1595a78" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.134996 1049 raft_consensus.cc:2463] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.136075 1218 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.008s sys 0.001s
I20250112 20:51:26.136511 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c31c29073aa46ffa847905be1595a78" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.136761 776 raft_consensus.cc:2463] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.139940 1295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:26.141916 1218 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250112 20:51:26.142082 1218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
uuid: "b985fd656ce44131bc91ccfeb79587c5"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:26.143257 1218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:26.145840 1271 raft_consensus.cc:2743] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.146462 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32dc02497f64a2e9ad99da6c9115d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.148711 847 leader_election.cc:304] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.155681 1271 raft_consensus.cc:2798] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.156270 1186 ts_tablet_manager.cc:1905] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.157886 1178 tablet_bootstrap.cc:654] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.158620 1271 raft_consensus.cc:491] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.158562 1169 raft_consensus.cc:513] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.158771 1271 raft_consensus.cc:3054] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.159543 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a8f71233d04ca39e49932308e56492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.166528 1164 tablet_bootstrap.cc:654] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.174659 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e040383f6e2a4ccf9614fd2d119a3cb4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:26.174952 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a90f801d854d4d8f7f09112e2d2929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.174950 1049 raft_consensus.cc:3054] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.178372 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e040383f6e2a4ccf9614fd2d119a3cb4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.178637 911 raft_consensus.cc:3054] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.178817 1049 raft_consensus.cc:2463] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:26.179236 710 leader_election.cc:304] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.179530 1228 raft_consensus.cc:491] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.179559 1227 raft_consensus.cc:2798] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.181042 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e2f2bcbfc0194e0e9342fe49a37abec0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.181667 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.181717 1228 raft_consensus.cc:513] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.182075 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cc561e5052ca4268b3733b33a981bf86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.182610 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc561e5052ca4268b3733b33a981bf86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.186254 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.186712 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.190129 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9cf0195d130e459fbc8e651f57546be7" } modify_external_catalogs: true
I20250112 20:51:26.190366 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9cf0195d130e459fbc8e651f57546be7" } modify_external_catalogs: true
I20250112 20:51:26.190614 1303 raft_consensus.cc:491] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.190769 1303 raft_consensus.cc:513] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.191506 1303 leader_election.cc:290] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.191774 1301 raft_consensus.cc:491] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.192503 1301 raft_consensus.cc:513] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.192860 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.193728 1301 leader_election.cc:290] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.193656 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.193955 710 leader_election.cc:343] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.194485 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.195299 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.195628 710 leader_election.cc:343] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:26.195813 710 leader_election.cc:343] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.195892 710 leader_election.cc:304] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.196381 1227 raft_consensus.cc:695] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:26.196697 1227 consensus_queue.cc:237] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.198313 1218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:26.198657 911 raft_consensus.cc:2463] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:26.199409 1218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20250112 20:51:26.199640 710 leader_election.cc:343] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.199709 1218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:26.199751 710 leader_election.cc:304] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.199991 1301 raft_consensus.cc:2743] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.200197 1301 raft_consensus.cc:2743] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.203788 1164 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.204052 1164 ts_tablet_manager.cc:1397] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.002s
I20250112 20:51:26.205454 1301 raft_consensus.cc:491] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.205118 1164 raft_consensus.cc:357] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.205627 1301 raft_consensus.cc:513] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.206012 1164 raft_consensus.cc:383] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.206163 1164 raft_consensus.cc:738] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.206301 1301 leader_election.cc:290] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.206607 1164 consensus_queue.cc:260] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:26.209383 710 leader_election.cc:343] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:26.210057 1228 leader_election.cc:290] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.210592 1169 leader_election.cc:290] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.212267 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.212347 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.212565 1049 raft_consensus.cc:2463] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.212596 776 raft_consensus.cc:2463] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.213215 847 leader_election.cc:304] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.213639 1228 raft_consensus.cc:2798] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.213745 1228 raft_consensus.cc:491] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.213876 1228 raft_consensus.cc:3054] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.215112 1164 ts_tablet_manager.cc:1428] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 20:51:26.215890 1164 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.217151 1228 raft_consensus.cc:513] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.218641 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.218910 776 raft_consensus.cc:3054] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.220674 1164 tablet_bootstrap.cc:654] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.222735 776 raft_consensus.cc:2463] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.236027 1164 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.236208 1164 ts_tablet_manager.cc:1397] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20250112 20:51:26.236671 847 leader_election.cc:304] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.237450 1307 raft_consensus.cc:2798] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.237406 1271 raft_consensus.cc:513] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.237411 1164 raft_consensus.cc:357] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.237856 1164 raft_consensus.cc:383] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.237973 1164 raft_consensus.cc:738] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.238339 1164 consensus_queue.cc:260] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.238998 1271 leader_election.cc:290] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:26.241194 710 leader_election.cc:343] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:26.241381 710 leader_election.cc:304] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.244547 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c31c29073aa46ffa847905be1595a78" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:26.244840 1049 raft_consensus.cc:3054] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.244825 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c31c29073aa46ffa847905be1595a78" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.245043 776 raft_consensus.cc:3054] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.245815 1307 raft_consensus.cc:695] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:26.246047 1307 consensus_queue.cc:237] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.248420 776 raft_consensus.cc:2463] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.248795 847 leader_election.cc:304] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.249107 1308 raft_consensus.cc:2798] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.249462 1308 raft_consensus.cc:695] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:26.250123 1227 raft_consensus.cc:2743] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.250360 1308 consensus_queue.cc:237] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.251040 1049 raft_consensus.cc:2463] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.253245 1164 ts_tablet_manager.cc:1428] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250112 20:51:26.257738 1228 leader_election.cc:290] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.258671 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07cef7306074ba3b7f8606bdfa6204a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:26.258998 1049 raft_consensus.cc:3054] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.261300 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9a692cc5fb6f4afc854c4821274edfdb
W20250112 20:51:26.262442 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d616a8ad9ad4777b44deb77bc321d66 consensus state has no leader
I20250112 20:51:26.263625 1049 raft_consensus.cc:2463] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.264232 1164 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.264250 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.264494 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:26.264897 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9a692cc5fb6f4afc854c4821274edfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a692cc5fb6f4afc854c4821274edfdb already in progress: creating tablet
W20250112 20:51:26.265517 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9a692cc5fb6f4afc854c4821274edfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a692cc5fb6f4afc854c4821274edfdb already in progress: creating tablet
I20250112 20:51:26.269165 1164 tablet_bootstrap.cc:654] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.274209 1164 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.274402 1164 ts_tablet_manager.cc:1397] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20250112 20:51:26.275568 1164 raft_consensus.cc:357] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.275900 1164 raft_consensus.cc:383] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.276007 1164 raft_consensus.cc:738] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.276346 1164 consensus_queue.cc:260] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.284476 1164 ts_tablet_manager.cc:1428] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20250112 20:51:26.289391 1178 tablet_bootstrap.cc:492] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.289821 1178 ts_tablet_manager.cc:1397] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.257s user 0.002s sys 0.003s
I20250112 20:51:26.290052 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.290656 1164 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.291113 1178 raft_consensus.cc:357] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.292449 1178 raft_consensus.cc:383] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.292546 1178 raft_consensus.cc:738] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.292896 1178 consensus_queue.cc:260] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.293120 1164 tablet_bootstrap.cc:654] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.294173 1168 raft_consensus.cc:491] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.294385 1168 raft_consensus.cc:513] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.294569 1178 ts_tablet_manager.cc:1428] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:26.295228 1168 leader_election.cc:290] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.295848 1178 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:26.296620 983 leader_election.cc:343] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e3a895feb0504d39ba8f93b25e2e032d
W20250112 20:51:26.297477 983 leader_election.cc:343] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e3a895feb0504d39ba8f93b25e2e032d
I20250112 20:51:26.297638 983 leader_election.cc:304] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.298208 1168 raft_consensus.cc:2743] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.300015 1178 tablet_bootstrap.cc:654] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.306032 1164 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.306190 1164 ts_tablet_manager.cc:1397] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20250112 20:51:26.307214 1164 raft_consensus.cc:357] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.307521 1164 raft_consensus.cc:383] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.307615 1164 raft_consensus.cc:738] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.307902 1164 consensus_queue.cc:260] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.311445 1178 tablet_bootstrap.cc:492] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.311563 1164 ts_tablet_manager.cc:1428] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250112 20:51:26.311717 1178 ts_tablet_manager.cc:1397] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.002s
I20250112 20:51:26.312568 1178 raft_consensus.cc:357] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.312445 1164 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.312873 1178 raft_consensus.cc:383] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.313063 1178 raft_consensus.cc:738] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.313357 1178 consensus_queue.cc:260] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.314366 1178 ts_tablet_manager.cc:1428] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:26.315011 1164 tablet_bootstrap.cc:654] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.315382 1178 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.317489 1178 tablet_bootstrap.cc:654] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.320423 1164 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.320587 1164 ts_tablet_manager.cc:1397] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20250112 20:51:26.321516 1164 raft_consensus.cc:357] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.321780 1164 raft_consensus.cc:383] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.321863 1164 raft_consensus.cc:738] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.322149 1164 consensus_queue.cc:260] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.323654 1164 ts_tablet_manager.cc:1428] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:26.324661 1164 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.325279 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_983423b832f34c34a969cb2ad8f6db40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.326001 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_983423b832f34c34a969cb2ad8f6db40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.326630 1164 tablet_bootstrap.cc:654] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.326642 1186 ts_tablet_manager.cc:1918] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:26.326752 1186 log.cc:1198] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/24abf89a9b2542d7bac2f7499698df3a
I20250112 20:51:26.327716 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9d07ceff6ea140e997aff9afbaf71e91" } modify_external_catalogs: true
I20250112 20:51:26.327886 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9d07ceff6ea140e997aff9afbaf71e91" } modify_external_catalogs: true
I20250112 20:51:26.328621 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e" } modify_external_catalogs: true
I20250112 20:51:26.328820 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e" } modify_external_catalogs: true
I20250112 20:51:26.328953 1164 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.329267 1164 ts_tablet_manager.cc:1397] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:26.330106 1164 raft_consensus.cc:357] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.330339 1164 raft_consensus.cc:383] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.330436 1164 raft_consensus.cc:738] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.330732 1164 consensus_queue.cc:260] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.332279 1164 ts_tablet_manager.cc:1428] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:26.333043 1164 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.336565 1164 tablet_bootstrap.cc:654] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.337466 1186 ts_tablet_manager.cc:1939] T 24abf89a9b2542d7bac2f7499698df3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.340315 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 24abf89a9b2542d7bac2f7499698df3a (table default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]) successfully deleted
W20250112 20:51:26.340684 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5ea7336d13ea46748167e35e7e7e8c98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ea7336d13ea46748167e35e7e7e8c98 already in progress: creating tablet
I20250112 20:51:26.344528 1163 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.344758 1163 ts_tablet_manager.cc:1397] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.222s user 0.039s sys 0.087s
I20250112 20:51:26.346109 1163 raft_consensus.cc:357] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.346261 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:26.346364 1163 raft_consensus.cc:383] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.347493 890 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.347517 751 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.347581 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.346434 1163 raft_consensus.cc:738] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.347806 1185 tablet_replica.cc:331] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.347992 1185 raft_consensus.cc:2238] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.347959 1163 consensus_queue.cc:260] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.348178 1185 raft_consensus.cc:2267] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:26.348294 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cc9aa4f42f89455cba7c72e7a49c6a94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9aa4f42f89455cba7c72e7a49c6a94 already in progress: creating tablet
I20250112 20:51:26.348492 1163 ts_tablet_manager.cc:1428] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:26.348924 1163 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.349139 1185 ts_tablet_manager.cc:1905] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.351456 1163 tablet_bootstrap.cc:654] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.356114 1185 ts_tablet_manager.cc:1918] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.356266 1185 log.cc:1198] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:26.356673 1185 ts_tablet_manager.cc:1939] T cc9aa4f42f89455cba7c72e7a49c6a94 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.357930 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet cc9aa4f42f89455cba7c72e7a49c6a94 (table default.Table_9cf0195d130e459fbc8e651f57546be7 [id=7bde50a83284433daf462fe09db9061c]) successfully deleted
I20250112 20:51:26.358777 1178 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.359005 1178 ts_tablet_manager.cc:1397] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20250112 20:51:26.359222 1168 raft_consensus.cc:491] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.359380 1168 raft_consensus.cc:513] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.359810 1178 raft_consensus.cc:357] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.360044 1178 raft_consensus.cc:383] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.360033 1168 leader_election.cc:290] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.360128 1178 raft_consensus.cc:738] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:26.361212 983 leader_election.cc:343] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e482a2543cdf4020a03f131c980e7928
I20250112 20:51:26.362121 1178 consensus_queue.cc:260] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.363423 1178 ts_tablet_manager.cc:1428] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:26.370755 1308 raft_consensus.cc:491] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.370988 1308 raft_consensus.cc:513] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.371750 1308 leader_election.cc:290] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.373139 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.373193 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.373443 776 raft_consensus.cc:2463] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.373932 847 leader_election.cc:304] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.374059 1303 raft_consensus.cc:491] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.374197 1178 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:26.374187 983 leader_election.cc:343] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e482a2543cdf4020a03f131c980e7928
I20250112 20:51:26.374214 1303 raft_consensus.cc:513] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.374307 983 leader_election.cc:304] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:26.374219 847 leader_election.cc:343] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.374625 1308 raft_consensus.cc:2798] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.374711 1308 raft_consensus.cc:491] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.374768 1303 leader_election.cc:290] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.374809 1308 raft_consensus.cc:3054] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.375193 1168 raft_consensus.cc:2743] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.376055 1178 tablet_bootstrap.cc:654] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.375999 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a57ea399194a45bc8990a029550440" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.376224 1049 raft_consensus.cc:2463] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:26.376431 1186 tablet_replica.cc:331] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.377249 710 leader_election.cc:304] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.377647 1303 raft_consensus.cc:2798] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.377745 1303 raft_consensus.cc:491] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.377861 1303 raft_consensus.cc:3054] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.383100 1303 raft_consensus.cc:513] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.385655 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a57ea399194a45bc8990a029550440" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:26.385845 1049 raft_consensus.cc:3054] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.388280 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a078ad9ead144747a9f5af58eb338733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.389027 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a078ad9ead144747a9f5af58eb338733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.390656 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_884652a61536446f903e8a58dac9590f" } new_table_name: "default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981" modify_external_catalogs: true
I20250112 20:51:26.391921 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_42918b2644f14f05ae359f8a5c931c2a" } new_table_name: "default.Table_c6bfd98d1c8e429587688a4a79f4ec65" modify_external_catalogs: true
I20250112 20:51:26.391959 1049 raft_consensus.cc:2463] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:26.392452 710 leader_election.cc:304] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.392830 1227 raft_consensus.cc:2798] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.392982 1227 raft_consensus.cc:695] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:26.393303 1227 consensus_queue.cc:237] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.395332 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 9a692cc5fb6f4afc854c4821274edfdb from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.398547 1303 leader_election.cc:290] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.399291 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a57ea399194a45bc8990a029550440" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.400992 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ef89229192934339954566e366b6891c
I20250112 20:51:26.401989 1308 raft_consensus.cc:513] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.402438 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.402765 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:26.403415 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ef89229192934339954566e366b6891c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef89229192934339954566e366b6891c already in progress: creating tablet
I20250112 20:51:26.403831 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:26.404122 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ef89229192934339954566e366b6891c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef89229192934339954566e366b6891c already in progress: creating tablet
I20250112 20:51:26.405784 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.405999 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_75837bdca38e4a358cb5c44071b13b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.406551 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75837bdca38e4a358cb5c44071b13b26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:26.406633 847 leader_election.cc:343] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.407220 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3bd4eb92114bdb98b923fda9192cd6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.407424 776 raft_consensus.cc:3054] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.409595 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f1e1ad1b8b6a487ca4a40ae50884b72d" } new_table_name: "default.Table_9dfb4678229745ae83688748826dc594" modify_external_catalogs: true
I20250112 20:51:26.410270 776 raft_consensus.cc:2463] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.412431 911 raft_consensus.cc:2463] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:26.406427 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7a57ea399194a45bc8990a029550440" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.413908 1308 leader_election.cc:290] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.415143 913 raft_consensus.cc:3054] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.415673 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4fb0c297ce434ae4be2c6eba23a061a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.416126 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.418005 913 raft_consensus.cc:2463] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:26.418329 847 leader_election.cc:304] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:26.419222 1308 raft_consensus.cc:2798] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.419406 1308 raft_consensus.cc:695] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:26.419667 1308 consensus_queue.cc:237] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:26.429111 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0f1c76db0d64624bd43f0e538f0649d consensus state has no leader
W20250112 20:51:26.430264 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c55b922c39545ac9c89115cd587ea8c consensus state has no leader
I20250112 20:51:26.430637 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e3b1f099596246f78012a56ae87a63bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.431115 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3b1f099596246f78012a56ae87a63bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.435083 1186 raft_consensus.cc:2238] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.435281 1186 raft_consensus.cc:2267] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.435849 1186 ts_tablet_manager.cc:1905] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.437245 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0db67ae601df4b229f938deaeb0c6f8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.437747 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0db67ae601df4b229f938deaeb0c6f8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.440070 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 158cfe5ba52c4771af97841885b29f9d (DEFAULT_TABLE table=default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.440492 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158cfe5ba52c4771af97841885b29f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.440492 751 tablet_service.cc:1467] Processing CreateTablet for tablet 158cfe5ba52c4771af97841885b29f9d (DEFAULT_TABLE table=default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.440860 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158cfe5ba52c4771af97841885b29f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.441630 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 29b70ffb3e3341d398d36ff76993e9ac from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.443046 890 tablet_service.cc:1467] Processing CreateTablet for tablet 158cfe5ba52c4771af97841885b29f9d (DEFAULT_TABLE table=default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.443337 884 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4421d16e3f423794c44970a2469b10 (DEFAULT_TABLE table=default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.443425 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158cfe5ba52c4771af97841885b29f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.443674 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4421d16e3f423794c44970a2469b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.443964 887 tablet_service.cc:1467] Processing CreateTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 (DEFAULT_TABLE table=default.Table_22423ed5e6774aa6b41f72eb92b3ab3b [id=be8442e96b3b4635ba6dac3956ee9de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.444262 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbbefa7b55041a6a6b50b3c96c45833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.445048 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4421d16e3f423794c44970a2469b10 (DEFAULT_TABLE table=default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.445048 750 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4421d16e3f423794c44970a2469b10 (DEFAULT_TABLE table=default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.445382 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4421d16e3f423794c44970a2469b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.445744 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4421d16e3f423794c44970a2469b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.447896 749 tablet_service.cc:1467] Processing CreateTablet for tablet 304e1c8abe094bd99a99fd358b697f81 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.448218 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304e1c8abe094bd99a99fd358b697f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.449033 886 tablet_service.cc:1467] Processing CreateTablet for tablet 304e1c8abe094bd99a99fd358b697f81 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.449223 1028 tablet_service.cc:1467] Processing CreateTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 (DEFAULT_TABLE table=default.Table_22423ed5e6774aa6b41f72eb92b3ab3b [id=be8442e96b3b4635ba6dac3956ee9de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.449394 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304e1c8abe094bd99a99fd358b697f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.449550 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbbefa7b55041a6a6b50b3c96c45833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.449748 753 tablet_service.cc:1467] Processing CreateTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 (DEFAULT_TABLE table=default.Table_22423ed5e6774aa6b41f72eb92b3ab3b [id=be8442e96b3b4635ba6dac3956ee9de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.450125 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbbefa7b55041a6a6b50b3c96c45833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.449678 1303 raft_consensus.cc:491] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.451323 1303 raft_consensus.cc:513] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:26.451438 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet decdec0d34344ec4b422643b0d4e7659 consensus state has no leader
I20250112 20:51:26.451836 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d45f925e122a4a0d87d36700dddbdf92" } new_table_name: "default.Table_3061d11eaeb14b75a2d4147047f6e0c3" modify_external_catalogs: true
I20250112 20:51:26.451974 1303 leader_election.cc:290] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.452948 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.453619 710 leader_election.cc:343] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.455601 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 304e1c8abe094bd99a99fd358b697f81 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.455931 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304e1c8abe094bd99a99fd358b697f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.456439 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:26.456980 710 leader_election.cc:343] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.457104 710 leader_election.cc:304] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.457494 1303 raft_consensus.cc:2743] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.476953 1303 raft_consensus.cc:491] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.477180 1303 raft_consensus.cc:513] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.477938 1178 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.478058 1303 leader_election.cc:290] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.478106 1178 ts_tablet_manager.cc:1397] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.007s
I20250112 20:51:26.479403 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea7336d13ea46748167e35e7e7e8c98" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.480245 710 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.480640 1163 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.480648 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea7336d13ea46748167e35e7e7e8c98" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:26.481388 710 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.481429 1178 raft_consensus.cc:357] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.481624 1178 raft_consensus.cc:383] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.481575 710 leader_election.cc:304] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.481709 1178 raft_consensus.cc:738] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.481912 1178 consensus_queue.cc:260] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.482383 1178 ts_tablet_manager.cc:1428] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.482414 1303 raft_consensus.cc:2743] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.482681 1178 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.484331 1178 tablet_bootstrap.cc:654] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.484843 1163 ts_tablet_manager.cc:1397] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.132s user 0.012s sys 0.025s
I20250112 20:51:26.488209 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 719b906973a8490898d6befce49910d3 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.488337 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 29b70ffb3e3341d398d36ff76993e9ac from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:26.489832 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2a8f71233d04ca39e49932308e56492 consensus state has no leader
I20250112 20:51:26.490128 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.490747 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a078ad9ead144747a9f5af58eb338733" } modify_external_catalogs: true
I20250112 20:51:26.490980 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a078ad9ead144747a9f5af58eb338733" } modify_external_catalogs: true
I20250112 20:51:26.491894 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_639d154365b14495aee0bffa52ccb7f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.492561 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_639d154365b14495aee0bffa52ccb7f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.485783 1163 raft_consensus.cc:357] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.495851 1163 raft_consensus.cc:383] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.496014 1163 raft_consensus.cc:738] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.496300 1163 consensus_queue.cc:260] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.496631 1178 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.497251 1163 ts_tablet_manager.cc:1428] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:26.497555 1178 ts_tablet_manager.cc:1397] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20250112 20:51:26.498167 1164 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.498342 1164 ts_tablet_manager.cc:1397] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.165s user 0.025s sys 0.022s
I20250112 20:51:26.499055 1164 raft_consensus.cc:357] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.499271 1164 raft_consensus.cc:383] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.499347 1164 raft_consensus.cc:738] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.499536 1163 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.499577 1164 consensus_queue.cc:260] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.500109 1164 ts_tablet_manager.cc:1428] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.500401 1164 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.500507 1178 raft_consensus.cc:357] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.502072 1164 tablet_bootstrap.cc:654] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.503551 1178 raft_consensus.cc:383] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.504451 1178 raft_consensus.cc:738] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.504850 1178 consensus_queue.cc:260] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.506191 1163 tablet_bootstrap.cc:654] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.507707 1178 ts_tablet_manager.cc:1428] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:26.508051 1178 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.502039 1186 ts_tablet_manager.cc:1918] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.509740 1186 log.cc:1198] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/df9fcf5bc7584cdcb17b19fc69802b84
I20250112 20:51:26.510142 1186 ts_tablet_manager.cc:1939] T df9fcf5bc7584cdcb17b19fc69802b84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.511247 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:26.511577 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:26.514794 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet df9fcf5bc7584cdcb17b19fc69802b84 (table default.Table_cf7c4a1781c94c5a8562c48c3d8b84c1 [id=a578996c09484fb1a6b08eb55eff74d0]) successfully deleted
I20250112 20:51:26.515858 446 catalog_manager.cc:5526] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:26.517024 1186 tablet_replica.cc:331] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.517282 1186 raft_consensus.cc:2238] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.517462 1186 raft_consensus.cc:2267] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.517447 1168 raft_consensus.cc:491] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.512483 1178 tablet_bootstrap.cc:654] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.517604 1168 raft_consensus.cc:513] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.518308 1168 leader_election.cc:290] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.518441 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab5ecd2698b4952a892135b3593106a (DEFAULT_TABLE table=default.Table_983423b832f34c34a969cb2ad8f6db40 [id=75ce02402d924e7f8153b6348a26af9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.518504 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3ade9b410d8443249730cacdf7266eb2 (DEFAULT_TABLE table=default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.518860 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8c38c8594a384689ae57c26174765284 (DEFAULT_TABLE table=default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.518884 888 tablet_service.cc:1467] Processing CreateTablet for tablet 44f11304909b416aa1e9db1abf633a89 (DEFAULT_TABLE table=default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73 [id=14ecb9204295481eb572f350eb539e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.519385 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.519520 893 tablet_service.cc:1467] Processing CreateTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f (DEFAULT_TABLE table=default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.520051 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ade9b410d8443249730cacdf7266eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.520300 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ecd2698b4952a892135b3593106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.521855 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c7a54ce78854137ab2d12e9ce055565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.523910 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8c38c8594a384689ae57c26174765284 (DEFAULT_TABLE table=default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.524312 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c38c8594a384689ae57c26174765284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.524379 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab5ecd2698b4952a892135b3593106a (DEFAULT_TABLE table=default.Table_983423b832f34c34a969cb2ad8f6db40 [id=75ce02402d924e7f8153b6348a26af9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.524701 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ecd2698b4952a892135b3593106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.529083 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.529402 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c7a54ce78854137ab2d12e9ce055565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.530360 1164 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.530510 1164 ts_tablet_manager.cc:1397] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.009s
I20250112 20:51:26.530593 1163 tablet_bootstrap.cc:492] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.530889 1163 ts_tablet_manager.cc:1397] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20250112 20:51:26.531474 1164 raft_consensus.cc:357] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.531810 1164 raft_consensus.cc:383] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.531992 1164 raft_consensus.cc:738] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.532212 1164 consensus_queue.cc:260] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.532975 1163 raft_consensus.cc:357] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.533622 1163 raft_consensus.cc:383] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.534032 1163 raft_consensus.cc:738] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.532760 1164 ts_tablet_manager.cc:1428] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.536322 1164 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.532842 752 tablet_service.cc:1467] Processing CreateTablet for tablet 44f11304909b416aa1e9db1abf633a89 (DEFAULT_TABLE table=default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73 [id=14ecb9204295481eb572f350eb539e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.536901 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f11304909b416aa1e9db1abf633a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.538198 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c38c8594a384689ae57c26174765284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.540449 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f11304909b416aa1e9db1abf633a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.541107 1186 ts_tablet_manager.cc:1905] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.541395 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc5af0bc83bc415b972cbbe0fb39f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.541766 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8c38c8594a384689ae57c26174765284 (DEFAULT_TABLE table=default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.542068 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3ade9b410d8443249730cacdf7266eb2 (DEFAULT_TABLE table=default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.542162 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c38c8594a384689ae57c26174765284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.542388 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ade9b410d8443249730cacdf7266eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.542615 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab5ecd2698b4952a892135b3593106a (DEFAULT_TABLE table=default.Table_983423b832f34c34a969cb2ad8f6db40 [id=75ce02402d924e7f8153b6348a26af9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.542972 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab5ecd2698b4952a892135b3593106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.545289 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f (DEFAULT_TABLE table=default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.545599 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc5af0bc83bc415b972cbbe0fb39f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.546532 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.546871 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c7a54ce78854137ab2d12e9ce055565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.549017 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 44f11304909b416aa1e9db1abf633a89 (DEFAULT_TABLE table=default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73 [id=14ecb9204295481eb572f350eb539e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.549355 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f11304909b416aa1e9db1abf633a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.548918 1163 consensus_queue.cc:260] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.550343 1168 raft_consensus.cc:491] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.550500 1168 raft_consensus.cc:513] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.551776 1168 leader_election.cc:290] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.554982 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:26.557957 983 leader_election.cc:343] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e814b7a9b4cb4336b62b7cd7e35461b6
W20250112 20:51:26.558158 983 leader_election.cc:343] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:26.568048 1164 tablet_bootstrap.cc:654] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.570739 754 tablet_service.cc:1467] Processing CreateTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f (DEFAULT_TABLE table=default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.571728 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc5af0bc83bc415b972cbbe0fb39f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.572139 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:26.573333 983 leader_election.cc:343] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e814b7a9b4cb4336b62b7cd7e35461b6
I20250112 20:51:26.573500 983 leader_election.cc:304] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.573905 1168 raft_consensus.cc:2743] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.577389 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:26.577875 454 catalog_manager.cc:6596] ReplaceTablet: tablet 29b70ffb3e3341d398d36ff76993e9ac deleted and replaced by tablet d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:26.578459 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.578881 776 raft_consensus.cc:2463] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.579123 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 158cfe5ba52c4771af97841885b29f9d
W20250112 20:51:26.579185 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 29b70ffb3e3341d398d36ff76993e9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b70ffb3e3341d398d36ff76993e9ac already in progress: creating tablet
I20250112 20:51:26.580164 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 719b906973a8490898d6befce49910d3
I20250112 20:51:26.580166 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.580554 455 catalog_manager.cc:6596] ReplaceTablet: tablet 719b906973a8490898d6befce49910d3 deleted and replaced by tablet 271b8776c774488089ab1c794dbf823e
I20250112 20:51:26.581034 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 719b906973a8490898d6befce49910d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 271b8776c774488089ab1c794dbf823e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.581058 983 leader_election.cc:304] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:26.581151 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 158cfe5ba52c4771af97841885b29f9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158cfe5ba52c4771af97841885b29f9d already in progress: creating tablet
I20250112 20:51:26.581357 1168 raft_consensus.cc:2798] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.581422 1168 raft_consensus.cc:491] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.581525 1168 raft_consensus.cc:3054] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.583521 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet eb1dffbe43ba4eeab7ee12d901b62ffb (table default.Table_20471004d83e485dbea6520c7bdb37b0 [id=1d1adf4313fb4c69803023132dd314c2]). Expected version 3 got 0
I20250112 20:51:26.583768 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:26.583859 1168 raft_consensus.cc:513] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.584120 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:26.585213 891 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.585575 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.585806 1168 leader_election.cc:290] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.586010 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 719b906973a8490898d6befce49910d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 271b8776c774488089ab1c794dbf823e) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:26.586508 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 719b906973a8490898d6befce49910d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 719b906973a8490898d6befce49910d3 already in progress: creating tablet
I20250112 20:51:26.586508 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:26.586925 983 leader_election.cc:343] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:26.587080 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 with delete_type TABLET_DATA_DELETED (replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.587188 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7f39195945024f4fba9942faec70bc31
I20250112 20:51:26.588366 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.591523 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet e3a895feb0504d39ba8f93b25e2e032d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ad0df4285db4bb6954f3e4c8d858115) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.591642 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.591734 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e482a2543cdf4020a03f131c980e7928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.591830 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet e814b7a9b4cb4336b62b7cd7e35461b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.592768 1184 tablet_replica.cc:331] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.593008 1184 raft_consensus.cc:2238] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.593150 1184 raft_consensus.cc:2267] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.593713 1184 ts_tablet_manager.cc:1905] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.596241 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.596479 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:26.598629 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7f39195945024f4fba9942faec70bc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f39195945024f4fba9942faec70bc31 already in progress: creating tablet
I20250112 20:51:26.600346 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3ade9b410d8443249730cacdf7266eb2 (DEFAULT_TABLE table=default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.600596 1184 ts_tablet_manager.cc:1918] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.600802 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ade9b410d8443249730cacdf7266eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.600939 1184 log.cc:1198] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e3a895feb0504d39ba8f93b25e2e032d
I20250112 20:51:26.601100 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.601529 1184 ts_tablet_manager.cc:1939] T e3a895feb0504d39ba8f93b25e2e032d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.601389 1308 raft_consensus.cc:491] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.601717 1308 raft_consensus.cc:513] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.601898 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.602136 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 719b906973a8490898d6befce49910d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 271b8776c774488089ab1c794dbf823e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.602419 1308 leader_election.cc:290] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.602550 1185 tablet_replica.cc:331] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.603529 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.603533 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.603750 1049 raft_consensus.cc:2463] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.604159 847 leader_election.cc:304] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.604434 1308 raft_consensus.cc:2798] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.604552 1308 raft_consensus.cc:491] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.604637 1308 raft_consensus.cc:3054] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.604984 1185 raft_consensus.cc:2238] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.605211 1185 raft_consensus.cc:2267] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.606819 1185 ts_tablet_manager.cc:1905] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.607884 1308 raft_consensus.cc:513] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.608695 1308 leader_election.cc:290] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.609310 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:26.609499 1049 raft_consensus.cc:3054] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.609550 1184 tablet_replica.cc:331] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.609773 1186 ts_tablet_manager.cc:1918] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.609895 1186 log.cc:1198] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7df227e3389841259dc1420ec800e1c1
I20250112 20:51:26.610047 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e3a895feb0504d39ba8f93b25e2e032d (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:26.610255 1186 ts_tablet_manager.cc:1939] T 7df227e3389841259dc1420ec800e1c1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.611253 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7df227e3389841259dc1420ec800e1c1 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
I20250112 20:51:26.611745 1049 raft_consensus.cc:2463] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.611868 1184 raft_consensus.cc:2238] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.612083 847 leader_election.cc:304] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.612188 1184 raft_consensus.cc:2267] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.612403 1308 raft_consensus.cc:2798] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.612548 1308 raft_consensus.cc:695] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:26.613091 1308 consensus_queue.cc:237] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.613273 1184 ts_tablet_manager.cc:1905] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.613584 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.614295 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 with delete_type TABLET_DATA_DELETED (replaced by tablet 953d2ec8a7564c0bbcf0166487dd1746) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.615883 774 raft_consensus.cc:2463] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.616799 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.616957 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.617063 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.618068 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df227e3389841259dc1420ec800e1c1" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.618296 776 raft_consensus.cc:3054] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.620117 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_97ae975748324aa3a68452cc348b9e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.620657 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97ae975748324aa3a68452cc348b9e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.621680 1184 ts_tablet_manager.cc:1918] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.621743 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f79679bd64041e69e8c15369ec4acfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.621791 1184 log.cc:1198] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8c7a94b2e9be4b5db3dfd6129b03cd25
I20250112 20:51:26.621950 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.622013 1163 ts_tablet_manager.cc:1428] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.091s user 0.018s sys 0.002s
I20250112 20:51:26.622361 1163 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.615796 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7a94b2e9be4b5db3dfd6129b03cd25" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.622551 775 raft_consensus.cc:3054] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.622733 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.623754 1184 ts_tablet_manager.cc:1939] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.625025 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 (table default.Table_a267a3d65a5849dca2db5fbd6f68c182 [id=8bd86656bb82483ea8cd34a80fa2b843]) successfully deleted
I20250112 20:51:26.625357 1184 tablet_replica.cc:331] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.625488 776 raft_consensus.cc:2463] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.625581 1184 raft_consensus.cc:2238] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.625756 1163 tablet_bootstrap.cc:654] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.625931 983 leader_election.cc:304] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.626349 1184 raft_consensus.cc:2267] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.626689 1354 raft_consensus.cc:2798] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.626942 1184 ts_tablet_manager.cc:1905] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.628466 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet eb1dffbe43ba4eeab7ee12d901b62ffb (table default.Table_20471004d83e485dbea6520c7bdb37b0 [id=1d1adf4313fb4c69803023132dd314c2]). Expected version 3 got 0
I20250112 20:51:26.629540 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8beb027f4d5340d4b25629e73057420e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:26.629827 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fb2be6ce07824fe6a27363d4d6ff0106" } modify_external_catalogs: true
I20250112 20:51:26.629992 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fb2be6ce07824fe6a27363d4d6ff0106" } modify_external_catalogs: true
W20250112 20:51:26.630149 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8beb027f4d5340d4b25629e73057420e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.631332 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c6bfd98d1c8e429587688a4a79f4ec65" } new_table_name: "default.Table_03812fa225b44a6ab342958dd68ec4c2" modify_external_catalogs: true
I20250112 20:51:26.635033 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_20471004d83e485dbea6520c7bdb37b0" } modify_external_catalogs: true
I20250112 20:51:26.635198 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_20471004d83e485dbea6520c7bdb37b0" } modify_external_catalogs: true
I20250112 20:51:26.635512 1178 tablet_bootstrap.cc:492] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.635700 1178 ts_tablet_manager.cc:1397] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.004s
I20250112 20:51:26.636703 1168 raft_consensus.cc:491] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.636855 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.636893 1168 raft_consensus.cc:513] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.637576 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.637617 1168 leader_election.cc:290] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.637866 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.638119 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_86cd6df4a9b94b1587e5d8b668fab3bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:26.638244 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e53e67076c4a5a89f1e93fc168a997 with delete_type TABLET_DATA_DELETED (replaced by tablet 2afa5b0717df4f83a2b44efac12fc06b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.638465 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df227e3389841259dc1420ec800e1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 616b8aa8fd62486c978c275712064f77) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:26.638612 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86cd6df4a9b94b1587e5d8b668fab3bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.638653 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.638899 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b07cef7306074ba3b7f8606bdfa6204a with delete_type TABLET_DATA_DELETED (replaced by tablet ed45eb5cbd2e489c83c3d343ca1adcfc) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.639084 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 24abf89a9b2542d7bac2f7499698df3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.639144 775 raft_consensus.cc:2463] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.640514 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad0df4285db4bb6954f3e4c8d858115" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.641937 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:26.644184 983 leader_election.cc:343] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:26.645344 1164 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.645605 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4ffedc1a9fee40b4bf3a93f178099ceb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.646214 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ffedc1a9fee40b4bf3a93f178099ceb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:26.646797 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 29b70ffb3e3341d398d36ff76993e9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b70ffb3e3341d398d36ff76993e9ac already in progress: creating tablet
I20250112 20:51:26.647087 1164 ts_tablet_manager.cc:1397] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.008s
W20250112 20:51:26.647079 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 158cfe5ba52c4771af97841885b29f9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158cfe5ba52c4771af97841885b29f9d already in progress: creating tablet
I20250112 20:51:26.639349 755 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3bd4eb92114bdb98b923fda9192cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.650445 1164 raft_consensus.cc:357] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.651511 1164 raft_consensus.cc:383] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.651616 1164 raft_consensus.cc:738] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.652231 1164 consensus_queue.cc:260] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.653144 1164 ts_tablet_manager.cc:1428] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:26.653585 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbcad11f1934759804e51554cd05e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.654021 1164 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.655910 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_95ebb558399d410fb0df44b18fdb3603"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.657172 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95ebb558399d410fb0df44b18fdb3603 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.636483 1178 raft_consensus.cc:357] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.657879 1178 raft_consensus.cc:383] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.658003 1178 raft_consensus.cc:738] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.659044 1164 tablet_bootstrap.cc:654] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.659467 1178 consensus_queue.cc:260] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.659894 913 raft_consensus.cc:2463] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.661446 983 leader_election.cc:304] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.661907 1168 raft_consensus.cc:2798] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.662005 1168 raft_consensus.cc:491] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.662101 1168 raft_consensus.cc:3054] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.662695 1184 ts_tablet_manager.cc:1918] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.627031 1354 raft_consensus.cc:695] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:26.663372 1354 consensus_queue.cc:237] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.665712 1168 raft_consensus.cc:513] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.666366 1168 leader_election.cc:290] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.668279 1184 log.cc:1198] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e482a2543cdf4020a03f131c980e7928
I20250112 20:51:26.669248 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad0df4285db4bb6954f3e4c8d858115" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.669471 913 raft_consensus.cc:3054] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.670610 1185 ts_tablet_manager.cc:1918] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.671362 1178 ts_tablet_manager.cc:1428] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.036s user 0.001s sys 0.002s
I20250112 20:51:26.671707 1178 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.672061 913 raft_consensus.cc:2463] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.673172 1185 log.cc:1198] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/719b906973a8490898d6befce49910d3
I20250112 20:51:26.673777 563 master_runner.cc:186] Error getting master registration for 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111),
I20250112 20:51:26.674024 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 20:51:26.676496 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8c38c8594a384689ae57c26174765284
I20250112 20:51:26.677271 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a267a3d65a5849dca2db5fbd6f68c182" } new_table_name: "default.Table_40069cc52b264283ac3b145871a3b426" modify_external_catalogs: true
I20250112 20:51:26.677822 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.678079 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.678225 1185 ts_tablet_manager.cc:1939] T 719b906973a8490898d6befce49910d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.678566 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_015060e7afea41ba852c420195188ddf" } new_table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" modify_external_catalogs: true
I20250112 20:51:26.678819 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.679883 1303 raft_consensus.cc:491] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.680078 1303 raft_consensus.cc:513] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.680611 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 719b906973a8490898d6befce49910d3 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
I20250112 20:51:26.680650 1303 leader_election.cc:290] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.680981 1185 tablet_replica.cc:331] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.681190 1185 raft_consensus.cc:2238] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.681365 1185 raft_consensus.cc:2267] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.681748 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.681908 1185 ts_tablet_manager.cc:1905] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:26.682804 710 leader_election.cc:343] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.689055 1186 tablet_replica.cc:331] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.689291 1186 raft_consensus.cc:2238] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.689550 1186 pending_rounds.cc:62] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:26.689654 1186 pending_rounds.cc:69] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584993269768192 op_type: NO_OP noop_request { }
I20250112 20:51:26.689816 1186 raft_consensus.cc:2883] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.689903 1184 ts_tablet_manager.cc:1939] T e482a2543cdf4020a03f131c980e7928 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.689949 1186 raft_consensus.cc:2267] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.691138 1186 ts_tablet_manager.cc:1905] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.694568 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0218ef4723ae4dcd96e64a39634592cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.695199 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0218ef4723ae4dcd96e64a39634592cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.696316 1178 tablet_bootstrap.cc:654] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.701426 1184 tablet_replica.cc:331] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.701665 1184 raft_consensus.cc:2238] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.703096 1178 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.703265 1178 ts_tablet_manager.cc:1397] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250112 20:51:26.704046 1178 raft_consensus.cc:357] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.704257 1178 raft_consensus.cc:383] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.704321 1178 raft_consensus.cc:738] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.704444 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e482a2543cdf4020a03f131c980e7928 (table default.Table_5595598433be4c89aa8aa789a148ab47 [id=497f2178f33046b7aeaae9c3d582ac39]) successfully deleted
I20250112 20:51:26.704546 1178 consensus_queue.cc:260] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.704751 1184 raft_consensus.cc:2267] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.705296 1184 ts_tablet_manager.cc:1905] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.706302 1178 ts_tablet_manager.cc:1428] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:26.706594 1178 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:26.707185 983 leader_election.cc:343] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:26.707316 1303 raft_consensus.cc:491] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.707504 1303 raft_consensus.cc:513] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.708181 1303 leader_election.cc:290] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.708374 1178 tablet_bootstrap.cc:654] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.708838 983 leader_election.cc:304] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.708847 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296703eedc6d459880a74b38e642bceb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.709247 1168 raft_consensus.cc:2798] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:26.709544 710 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.709851 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.710142 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.710685 710 consensus_peers.cc:487] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f79679bd64041e69e8c15369ec4acfb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:26.711697 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296703eedc6d459880a74b38e642bceb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.712107 710 leader_election.cc:343] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.712239 710 leader_election.cc:304] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.712312 1185 ts_tablet_manager.cc:1918] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.712445 1185 log.cc:1198] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8c7a94b2e9be4b5db3dfd6129b03cd25
I20250112 20:51:26.712512 1303 raft_consensus.cc:2743] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:26.713542 710 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.713685 710 leader_election.cc:304] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.714067 1303 raft_consensus.cc:2743] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.714522 1185 ts_tablet_manager.cc:1939] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:26.716099 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c55b922c39545ac9c89115cd587ea8c consensus state has no leader
I20250112 20:51:26.717411 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8c7a94b2e9be4b5db3dfd6129b03cd25 (table default.Table_a267a3d65a5849dca2db5fbd6f68c182 [id=8bd86656bb82483ea8cd34a80fa2b843]) successfully deleted
W20250112 20:51:26.718083 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 7f39195945024f4fba9942faec70bc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f39195945024f4fba9942faec70bc31 already in progress: creating tablet
I20250112 20:51:26.718389 1185 tablet_replica.cc:331] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.718631 1185 raft_consensus.cc:2238] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.718809 1185 raft_consensus.cc:2267] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.719060 1354 raft_consensus.cc:491] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.719216 1354 raft_consensus.cc:513] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.719841 1354 leader_election.cc:290] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.722283 1185 ts_tablet_manager.cc:1905] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.722501 1164 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.722641 1164 ts_tablet_manager.cc:1397] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.008s
W20250112 20:51:26.723407 983 leader_election.cc:343] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 24abf89a9b2542d7bac2f7499698df3a
I20250112 20:51:26.723491 1164 raft_consensus.cc:357] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.723718 1164 raft_consensus.cc:383] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.723827 1164 raft_consensus.cc:738] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.724066 1164 consensus_queue.cc:260] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.724644 1164 ts_tablet_manager.cc:1428] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:26.724566 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24abf89a9b2542d7bac2f7499698df3a" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.725982 775 raft_consensus.cc:2388] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:26.726032 1163 tablet_bootstrap.cc:492] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.726168 1163 ts_tablet_manager.cc:1397] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.002s
I20250112 20:51:26.726631 983 leader_election.cc:304] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.727063 1163 raft_consensus.cc:357] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.727258 1163 raft_consensus.cc:383] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.727353 1163 raft_consensus.cc:738] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.727567 1163 consensus_queue.cc:260] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.728173 1163 ts_tablet_manager.cc:1428] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:26.728477 1163 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.730572 1163 tablet_bootstrap.cc:654] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.731670 1164 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.733443 1164 tablet_bootstrap.cc:654] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.740005 1186 ts_tablet_manager.cc:1918] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:26.740149 1186 log.cc:1198] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b07cef7306074ba3b7f8606bdfa6204a
I20250112 20:51:26.740550 1186 ts_tablet_manager.cc:1939] T b07cef7306074ba3b7f8606bdfa6204a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.743115 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.743191 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c7a54ce78854137ab2d12e9ce055565 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.744695 1178 tablet_bootstrap.cc:492] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.744875 1178 ts_tablet_manager.cc:1397] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20250112 20:51:26.745697 1178 raft_consensus.cc:357] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.745893 1178 raft_consensus.cc:383] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.745992 1178 raft_consensus.cc:738] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.746253 1178 consensus_queue.cc:260] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.747501 1178 ts_tablet_manager.cc:1428] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:26.748893 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet eb1dffbe43ba4eeab7ee12d901b62ffb
I20250112 20:51:26.748947 1354 raft_consensus.cc:3054] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.750236 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_598b0b1981fc4153baa6e9b8b2521981" } new_table_name: "default.Table_4ff8c0d47d9e4525a7fa3b9304dc109e" modify_external_catalogs: true
I20250112 20:51:26.750392 884 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.750991 749 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:26.751091 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1dffbe43ba4eeab7ee12d901b62ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.751955 1185 ts_tablet_manager.cc:1918] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.752101 1185 log.cc:1198] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b07cef7306074ba3b7f8606bdfa6204a
I20250112 20:51:26.752446 1185 ts_tablet_manager.cc:1939] T b07cef7306074ba3b7f8606bdfa6204a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.753166 1184 ts_tablet_manager.cc:1918] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.753297 1184 log.cc:1198] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e814b7a9b4cb4336b62b7cd7e35461b6
I20250112 20:51:26.753417 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b07cef7306074ba3b7f8606bdfa6204a (table default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:26.753718 1184 ts_tablet_manager.cc:1939] T e814b7a9b4cb4336b62b7cd7e35461b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.754817 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e814b7a9b4cb4336b62b7cd7e35461b6 (table default.Table_5bfc59824f23459cade39fdb87c63491 [id=8ffaa9b9ac214c9f9d80dcc8865287a8]) successfully deleted
I20250112 20:51:26.755326 1184 tablet_replica.cc:331] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.757069 1185 tablet_replica.cc:331] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.757272 1185 raft_consensus.cc:2238] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.757472 1185 raft_consensus.cc:2267] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.758126 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_be025c70ef21488baea67cc8bd9c2d3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.758711 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be025c70ef21488baea67cc8bd9c2d3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.760239 1185 ts_tablet_manager.cc:1905] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.763236 1354 raft_consensus.cc:2743] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.769667 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9e27bc50d3ea443ab9b522fe653d2269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.770166 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e27bc50d3ea443ab9b522fe653d2269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.771944 1303 raft_consensus.cc:491] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.772132 1303 raft_consensus.cc:513] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.772894 1303 leader_election.cc:290] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.766299 1185 ts_tablet_manager.cc:1918] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.773188 1185 log.cc:1198] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7df227e3389841259dc1420ec800e1c1
I20250112 20:51:26.777168 1308 raft_consensus.cc:491] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.777365 1308 raft_consensus.cc:513] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.778049 1308 leader_election.cc:290] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.779089 1303 raft_consensus.cc:491] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.779242 1303 raft_consensus.cc:513] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.779597 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.779816 775 raft_consensus.cc:2463] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:26.779949 1303 leader_election.cc:290] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.780329 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8beb027f4d5340d4b25629e73057420e" } new_table_name: "default.Table_56e32a726bf14d0193beb81aef87a2de" modify_external_catalogs: true
W20250112 20:51:26.780890 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9c55b922c39545ac9c89115cd587ea8c consensus state has no leader
I20250112 20:51:26.781040 847 leader_election.cc:304] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.782210 1308 raft_consensus.cc:2798] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.782318 1308 raft_consensus.cc:491] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.782442 1308 raft_consensus.cc:3054] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.784989 1308 raft_consensus.cc:513] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.785660 1308 leader_election.cc:290] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.786160 1185 ts_tablet_manager.cc:1939] T 7df227e3389841259dc1420ec800e1c1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.786621 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.786825 775 raft_consensus.cc:3054] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:26.786954 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 953d2ec8a7564c0bbcf0166487dd1746 consensus state has no leader
I20250112 20:51:26.787463 1185 tablet_replica.cc:331] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.787782 1185 raft_consensus.cc:2238] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.788116 1185 pending_rounds.cc:62] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:26.788264 1185 pending_rounds.cc:69] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584991344324608 op_type: NO_OP noop_request { }
I20250112 20:51:26.788456 1185 raft_consensus.cc:2883] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.788655 1185 raft_consensus.cc:2267] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.789603 1185 ts_tablet_manager.cc:1905] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.789960 775 raft_consensus.cc:2463] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:26.790483 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_43c23610617d4b979805a95633d6c29c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.791013 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43c23610617d4b979805a95633d6c29c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.791828 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7df227e3389841259dc1420ec800e1c1 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
I20250112 20:51:26.792332 847 leader_election.cc:304] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:26.792673 1308 raft_consensus.cc:2798] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.792845 1308 raft_consensus.cc:695] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:26.793107 1308 consensus_queue.cc:237] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.773562 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.794610 710 leader_election.cc:343] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.777647 1168 raft_consensus.cc:695] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:26.778882 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.781582 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45eb5cbd2e489c83c3d343ca1adcfc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.786043 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:26.796373 710 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:26.796644 847 leader_election.cc:343] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:26.796980 847 leader_election.cc:343] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.797608 1168 consensus_queue.cc:237] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.799366 1185 ts_tablet_manager.cc:1918] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:26.802636 1163 tablet_bootstrap.cc:492] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:26.802819 1163 ts_tablet_manager.cc:1397] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.003s
I20250112 20:51:26.803591 1163 raft_consensus.cc:357] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.803834 1163 raft_consensus.cc:383] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.803931 1163 raft_consensus.cc:738] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.804193 1163 consensus_queue.cc:260] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.802241 1164 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.804507 1185 log.cc:1198] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/6f79679bd64041e69e8c15369ec4acfb
I20250112 20:51:26.804553 1164 ts_tablet_manager.cc:1397] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.004s
I20250112 20:51:26.805135 1185 ts_tablet_manager.cc:1939] T 6f79679bd64041e69e8c15369ec4acfb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.811470 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 6f79679bd64041e69e8c15369ec4acfb (table default.Table_9b7759bef3d2447596dfe1da9f6e5e72 [id=c56fb8c96e1b48c0a16bec40c1be564e]) successfully deleted
I20250112 20:51:26.811870 1185 tablet_replica.cc:331] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.812031 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 304e1c8abe094bd99a99fd358b697f81 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.812247 1185 raft_consensus.cc:2238] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.812398 1185 raft_consensus.cc:2267] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.805651 1164 raft_consensus.cc:357] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.812762 1164 raft_consensus.cc:383] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.812920 1164 raft_consensus.cc:738] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.813244 1164 consensus_queue.cc:260] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.813798 1164 ts_tablet_manager.cc:1428] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250112 20:51:26.814319 1164 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.814491 1184 raft_consensus.cc:2238] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.814996 1184 pending_rounds.cc:62] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:26.815104 1184 pending_rounds.cc:69] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584995528237056 op_type: NO_OP noop_request { }
I20250112 20:51:26.815248 1184 raft_consensus.cc:2883] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.815378 1184 raft_consensus.cc:2267] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.818048 1164 tablet_bootstrap.cc:654] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.818279 1354 raft_consensus.cc:491] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.818465 1354 raft_consensus.cc:513] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.819260 1354 leader_election.cc:290] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.821015 1184 ts_tablet_manager.cc:1905] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:26.821563 983 leader_election.cc:343] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 30e53e67076c4a5a89f1e93fc168a997
I20250112 20:51:26.827477 1184 ts_tablet_manager.cc:1918] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:26.827636 1184 log.cc:1198] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:26.828111 1184 ts_tablet_manager.cc:1939] T 8ad0df4285db4bb6954f3e4c8d858115 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.829886 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.830109 884 tablet_service.cc:1467] Processing CreateTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.830389 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3669355bc4b6bbc665edaa80c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.830547 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2235e2a63c4e1192b3260085908445 (DEFAULT_TABLE table=default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.830917 890 tablet_service.cc:1467] Processing CreateTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb (DEFAULT_TABLE table=default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.834278 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8ad0df4285db4bb6954f3e4c8d858115 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:26.834686 1184 tablet_replica.cc:331] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.835275 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 (DEFAULT_TABLE table=default.Table_0db67ae601df4b229f938deaeb0c6f8c [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.835314 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2235e2a63c4e1192b3260085908445 (DEFAULT_TABLE table=default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.835318 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.835685 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45eb5cbd2e489c83c3d343ca1adcfc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.835721 886 tablet_service.cc:1467] Processing CreateTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 (DEFAULT_TABLE table=default.Table_0db67ae601df4b229f938deaeb0c6f8c [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.836254 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e53e67076c4a5a89f1e93fc168a997" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.836503 910 raft_consensus.cc:2463] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.831312 749 tablet_service.cc:1467] Processing CreateTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.837492 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3669355bc4b6bbc665edaa80c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.837630 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba86d0ef63d94a55adf79bb136ea89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.841049 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb (DEFAULT_TABLE table=default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.841800 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2235e2a63c4e1192b3260085908445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.842377 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d3669355bc4b6bbc665edaa80c7a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.831666 751 tablet_service.cc:1467] Processing CreateTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 (DEFAULT_TABLE table=default.Table_0db67ae601df4b229f938deaeb0c6f8c [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.845104 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba86d0ef63d94a55adf79bb136ea89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.832023 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2235e2a63c4e1192b3260085908445 (DEFAULT_TABLE table=default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.849071 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2235e2a63c4e1192b3260085908445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.832341 753 tablet_service.cc:1467] Processing CreateTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb (DEFAULT_TABLE table=default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.812973 1185 ts_tablet_manager.cc:1905] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.853864 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a9e6cf325d48cc9a2a956400a68deb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:26.836894 710 leader_election.cc:343] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.854195 1184 raft_consensus.cc:2238] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.854313 710 leader_election.cc:304] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.854401 1184 raft_consensus.cc:2267] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:26.854928 710 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.855093 710 leader_election.cc:304] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.855754 1227 raft_consensus.cc:2743] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.856003 1227 raft_consensus.cc:2743] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.854225 983 leader_election.cc:304] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:26.856758 1168 raft_consensus.cc:2798] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.856902 1168 raft_consensus.cc:491] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.857057 1168 raft_consensus.cc:3054] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.860109 1185 ts_tablet_manager.cc:1918] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.860836 1184 ts_tablet_manager.cc:1905] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.861039 1185 log.cc:1198] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/313a5c9905bc445893f1ffa8752ca589
I20250112 20:51:26.861454 1185 ts_tablet_manager.cc:1939] T 313a5c9905bc445893f1ffa8752ca589 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.861147 1168 raft_consensus.cc:513] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.861415 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba86d0ef63d94a55adf79bb136ea89e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.862382 1168 leader_election.cc:290] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.862545 1168 raft_consensus.cc:491] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.862707 1168 raft_consensus.cc:513] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.863276 1168 leader_election.cc:290] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.861680 1303 raft_consensus.cc:491] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.864912 1303 raft_consensus.cc:513] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.865483 1303 leader_election.cc:290] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.866467 1185 tablet_replica.cc:331] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.866542 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e53e67076c4a5a89f1e93fc168a997" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.866639 1185 raft_consensus.cc:2238] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.866794 1185 raft_consensus.cc:2267] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.866781 910 raft_consensus.cc:3054] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.867278 1185 ts_tablet_manager.cc:1905] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.869035 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.869271 913 raft_consensus.cc:2463] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.870527 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 313a5c9905bc445893f1ffa8752ca589 (table default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]) successfully deleted
I20250112 20:51:26.871318 983 leader_election.cc:304] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.871810 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2235e2a63c4e1192b3260085908445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.876210 1354 raft_consensus.cc:2798] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.876330 1354 raft_consensus.cc:491] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.876410 1354 raft_consensus.cc:3054] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.876909 1184 ts_tablet_manager.cc:1918] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.877060 1184 log.cc:1198] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/aa3bd4eb92114bdb98b923fda9192cd6
I20250112 20:51:26.877063 1375 raft_consensus.cc:491] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.877349 1375 raft_consensus.cc:513] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.878386 1178 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.878472 1375 leader_election.cc:290] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.878983 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a9e6cf325d48cc9a2a956400a68deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.880074 1178 tablet_bootstrap.cc:654] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.881031 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49a9e6cf325d48cc9a2a956400a68deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.886652 1185 ts_tablet_manager.cc:1918] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.886771 1185 log.cc:1198] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/24abf89a9b2542d7bac2f7499698df3a
I20250112 20:51:26.886637 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.886934 913 raft_consensus.cc:2463] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.887115 1185 ts_tablet_manager.cc:1939] T 24abf89a9b2542d7bac2f7499698df3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.887691 1184 ts_tablet_manager.cc:1939] T aa3bd4eb92114bdb98b923fda9192cd6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.887724 983 leader_election.cc:304] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.888101 1375 raft_consensus.cc:2798] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.888211 1375 raft_consensus.cc:491] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.888299 1375 raft_consensus.cc:3054] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.888789 1184 tablet_replica.cc:331] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.888989 1184 raft_consensus.cc:2238] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.889223 1184 raft_consensus.cc:2267] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.889900 1184 ts_tablet_manager.cc:1905] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.892009 1375 raft_consensus.cc:513] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.894161 1303 raft_consensus.cc:491] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.894251 1163 ts_tablet_manager.cc:1428] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.091s user 0.001s sys 0.002s
I20250112 20:51:26.894315 1303 raft_consensus.cc:513] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.894577 1163 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:26.895012 1303 leader_election.cc:290] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.895278 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.895498 776 raft_consensus.cc:2463] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.895797 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:26.896054 774 raft_consensus.cc:2463] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:26.897138 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.899786 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.900341 1185 tablet_replica.cc:331] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.900588 1185 raft_consensus.cc:2238] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.901046 1185 raft_consensus.cc:2267] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:26.901556 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d770e0399e80491f8de9b08d1eb2f139 has no consensus state
I20250112 20:51:26.901778 1185 ts_tablet_manager.cc:1905] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.902347 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 24abf89a9b2542d7bac2f7499698df3a (table default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]) successfully deleted
W20250112 20:51:26.902743 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8ad0df4285db4bb6954f3e4c8d858115 because the tablet was not found. No further retry: Not found: Tablet not found: 8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:26.903156 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b07cef7306074ba3b7f8606bdfa6204a (table default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
W20250112 20:51:26.903460 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9a692cc5fb6f4afc854c4821274edfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a692cc5fb6f4afc854c4821274edfdb already in progress: creating tablet
I20250112 20:51:26.903698 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet aa3bd4eb92114bdb98b923fda9192cd6 (table default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]) successfully deleted
I20250112 20:51:26.905575 1178 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.905828 1178 ts_tablet_manager.cc:1397] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20250112 20:51:26.906489 910 raft_consensus.cc:2463] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.906818 1178 raft_consensus.cc:357] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.907047 1178 raft_consensus.cc:383] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.907122 1178 raft_consensus.cc:738] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.907176 983 leader_election.cc:304] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.907377 1178 consensus_queue.cc:260] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:26.907465 983 raft_consensus.cc:2681] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 20:51:26.908394 710 leader_election.cc:343] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.908464 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_bc82df6f9ef0472db33f87ad8da54503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:26.908808 710 leader_election.cc:343] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.908950 1178 ts_tablet_manager.cc:1428] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:26.909009 710 leader_election.cc:304] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.909333 1303 raft_consensus.cc:2743] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:26.909642 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc82df6f9ef0472db33f87ad8da54503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.909968 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:26.910933 983 leader_election.cc:343] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 30e53e67076c4a5a89f1e93fc168a997
I20250112 20:51:26.912333 1184 ts_tablet_manager.cc:1918] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.912451 1184 log.cc:1198] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/30e53e67076c4a5a89f1e93fc168a997
I20250112 20:51:26.914098 1178 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.915532 1164 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:26.915675 1164 ts_tablet_manager.cc:1397] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.002s
I20250112 20:51:26.920859 1178 tablet_bootstrap.cc:654] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:26.922678 710 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.923491 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad0df4285db4bb6954f3e4c8d858115 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f39195945024f4fba9942faec70bc31) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:26.923686 1185 ts_tablet_manager.cc:1918] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:26.925745 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cc9aa4f42f89455cba7c72e7a49c6a94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9aa4f42f89455cba7c72e7a49c6a94 already in progress: creating tablet
W20250112 20:51:26.926092 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ef89229192934339954566e366b6891c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef89229192934339954566e366b6891c already in progress: creating tablet
I20250112 20:51:26.927119 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.928378 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86cd6df4a9b94b1587e5d8b668fab3bc" } modify_external_catalogs: true
I20250112 20:51:26.928547 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 382aefb4a5e34999b560a9237839b380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.929378 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86cd6df4a9b94b1587e5d8b668fab3bc" } modify_external_catalogs: true
I20250112 20:51:26.931461 1375 leader_election.cc:290] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:26.932325 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.932672 910 raft_consensus.cc:3054] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.940395 1186 tablet_replica.cc:331] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.913071 1354 raft_consensus.cc:513] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.941679 1354 leader_election.cc:290] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:26.940721 1186 raft_consensus.cc:2238] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.942081 1186 raft_consensus.cc:2267] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.942663 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:26.942953 913 raft_consensus.cc:3054] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.943074 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65e847e586ae4fc99dee908b16e66be2" } modify_external_catalogs: true
I20250112 20:51:26.943301 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65e847e586ae4fc99dee908b16e66be2" } modify_external_catalogs: true
W20250112 20:51:26.946208 710 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.946131 1164 raft_consensus.cc:357] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.946350 710 leader_election.cc:304] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.946410 1164 raft_consensus.cc:383] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.946497 1164 raft_consensus.cc:738] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.946769 1164 consensus_queue.cc:260] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:26.947139 1227 raft_consensus.cc:2743] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.947373 1164 ts_tablet_manager.cc:1428] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250112 20:51:26.947444 1227 raft_consensus.cc:491] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.947470 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382aefb4a5e34999b560a9237839b380" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.947695 1164 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:26.947609 1227 raft_consensus.cc:513] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.947917 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1dffbe43ba4eeab7ee12d901b62ffb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:26.948102 776 raft_consensus.cc:3054] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.948458 1227 leader_election.cc:290] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:26.949086 1186 ts_tablet_manager.cc:1905] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.949245 913 raft_consensus.cc:2463] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.949847 983 leader_election.cc:304] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.950841 776 raft_consensus.cc:2463] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.952951 1354 raft_consensus.cc:2798] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.953119 1354 raft_consensus.cc:695] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:26.946159 1185 log.cc:1198] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/382aefb4a5e34999b560a9237839b380
I20250112 20:51:26.953414 1354 consensus_queue.cc:237] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:26.953603 1185 ts_tablet_manager.cc:1939] T 382aefb4a5e34999b560a9237839b380 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.954463 1185 tablet_replica.cc:331] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.954532 1184 ts_tablet_manager.cc:1939] T 30e53e67076c4a5a89f1e93fc168a997 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.954669 1185 raft_consensus.cc:2238] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.954849 1185 raft_consensus.cc:2267] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.955430 1184 tablet_replica.cc:331] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:26.955659 1184 raft_consensus.cc:2238] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.955886 1184 pending_rounds.cc:62] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:26.955889 1185 ts_tablet_manager.cc:1905] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.955965 1184 pending_rounds.cc:69] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584995022512128 op_type: NO_OP noop_request { }
I20250112 20:51:26.956073 1184 raft_consensus.cc:2883] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.956175 1184 raft_consensus.cc:2267] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.956974 1184 ts_tablet_manager.cc:1905] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:26.957005 910 raft_consensus.cc:2463] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:26.957548 983 leader_election.cc:304] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:26.957952 1375 raft_consensus.cc:2798] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.958123 1375 raft_consensus.cc:695] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:26.958454 1375 consensus_queue.cc:237] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:26.960667 983 leader_election.cc:343] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:26.961416 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:26.964538 1184 ts_tablet_manager.cc:1918] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:26.964659 1184 log.cc:1198] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7df227e3389841259dc1420ec800e1c1
W20250112 20:51:26.965171 710 leader_election.cc:343] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.965240 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:26.965636 1184 ts_tablet_manager.cc:1939] T 7df227e3389841259dc1420ec800e1c1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:26.965963 1163 tablet_bootstrap.cc:654] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:26.965977 710 leader_election.cc:343] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:26.966112 710 leader_election.cc:304] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:26.966536 1303 raft_consensus.cc:2743] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:26.966679 1185 ts_tablet_manager.cc:1918] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.966812 1185 log.cc:1198] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/aa3bd4eb92114bdb98b923fda9192cd6
I20250112 20:51:26.967188 1185 ts_tablet_manager.cc:1939] T aa3bd4eb92114bdb98b923fda9192cd6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.968420 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 30e53e67076c4a5a89f1e93fc168a997 (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
I20250112 20:51:26.968971 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 382aefb4a5e34999b560a9237839b380 (table default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]) successfully deleted
I20250112 20:51:26.969221 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet aa3bd4eb92114bdb98b923fda9192cd6 (table default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]) successfully deleted
I20250112 20:51:26.969521 1185 tablet_replica.cc:331] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:26.969756 1185 raft_consensus.cc:2238] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:26.969918 1185 raft_consensus.cc:2267] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:26.970435 1185 ts_tablet_manager.cc:1905] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:26.974812 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 has no consensus state
I20250112 20:51:26.975171 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 953d2ec8a7564c0bbcf0166487dd1746 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.980470 1164 tablet_bootstrap.cc:654] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.982367 1178 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:26.982693 1178 ts_tablet_manager.cc:1397] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.019s
I20250112 20:51:26.983508 1178 raft_consensus.cc:357] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.983734 1178 raft_consensus.cc:383] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.983829 1178 raft_consensus.cc:738] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.984098 1178 consensus_queue.cc:260] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:26.985682 1178 ts_tablet_manager.cc:1428] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:26.988356 447 master_service.cc:946] ReplaceTablet: received request to replace tablet ed45eb5cbd2e489c83c3d343ca1adcfc from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:26.988452 1178 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:26.989686 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 304e1c8abe094bd99a99fd358b697f81
I20250112 20:51:26.990128 452 catalog_manager.cc:6596] ReplaceTablet: tablet 304e1c8abe094bd99a99fd358b697f81 deleted and replaced by tablet e44422f75a744d078cdf4f0997242e22
I20250112 20:51:26.990307 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.990494 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_40069cc52b264283ac3b145871a3b426" } modify_external_catalogs: true
I20250112 20:51:26.990694 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_40069cc52b264283ac3b145871a3b426" } modify_external_catalogs: true
I20250112 20:51:26.992551 1178 tablet_bootstrap.cc:654] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.988095 1186 ts_tablet_manager.cc:1918] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.992798 1186 log.cc:1198] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/719b906973a8490898d6befce49910d3
I20250112 20:51:26.993073 1185 ts_tablet_manager.cc:1918] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:26.993779 1185 log.cc:1198] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9fbcad11f1934759804e51554cd05e3d
I20250112 20:51:26.994100 1185 ts_tablet_manager.cc:1939] T 9fbcad11f1934759804e51554cd05e3d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:26.995066 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9fbcad11f1934759804e51554cd05e3d (table default.Table_9ff857d5fbad4e3dbe3e5afcd835918d [id=66d28d001c68462c9c45d7ebd079903d]) successfully deleted
I20250112 20:51:26.995311 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1c7a54ce78854137ab2d12e9ce055565
I20250112 20:51:26.995708 1186 ts_tablet_manager.cc:1939] T 719b906973a8490898d6befce49910d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:26.995774 453 catalog_manager.cc:6596] ReplaceTablet: tablet 1c7a54ce78854137ab2d12e9ce055565 deleted and replaced by tablet 7293510f40804e2aabccf92986489dab
I20250112 20:51:26.995951 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.996893 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 719b906973a8490898d6befce49910d3 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
I20250112 20:51:26.998482 1186 tablet_replica.cc:331] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:26.998684 1186 raft_consensus.cc:2238] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:26.998941 1186 pending_rounds.cc:62] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:26.999011 1186 pending_rounds.cc:69] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584993977155584 op_type: NO_OP noop_request { }
I20250112 20:51:26.999191 1186 raft_consensus.cc:2883] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:26.999284 886 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:26.999498 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d4f060e4fa3648d6943cc97b9b3b7f51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.000097 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.999310 1186 raft_consensus.cc:2267] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.000664 1178 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:27.000866 1178 ts_tablet_manager.cc:1397] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250112 20:51:27.001633 1178 raft_consensus.cc:357] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.001895 1178 raft_consensus.cc:383] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.001979 1178 raft_consensus.cc:738] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.002377 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e18e998a23124368951dc9ad456b0567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.002707 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c38c8594a384689ae57c26174765284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c38c8594a384689ae57c26174765284 already in progress: creating tablet
W20250112 20:51:27.002915 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e18e998a23124368951dc9ad456b0567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.003155 1185 tablet_replica.cc:331] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:27.003381 1185 raft_consensus.cc:2238] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:27.003568 1185 raft_consensus.cc:2267] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.004094 1185 ts_tablet_manager.cc:1905] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.006510 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.009267 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.009783 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 3f54c82c5aad4bafb410cbbad6f7ea78
I20250112 20:51:27.010588 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.010860 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.011955 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b32dc02497f64a2e9ad99da6c9115d5d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:27.011873 1178 consensus_queue.cc:260] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.013335 1303 consensus_queue.cc:1035] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.014621 1178 ts_tablet_manager.cc:1428] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 20:51:27.017697 1178 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:27.022305 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3ade9b410d8443249730cacdf7266eb2
I20250112 20:51:27.023712 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.023993 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.024976 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.025158 1178 tablet_bootstrap.cc:654] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.031291 1178 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:27.031436 1178 ts_tablet_manager.cc:1397] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250112 20:51:27.032275 1178 raft_consensus.cc:357] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.032480 1178 raft_consensus.cc:383] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.032496 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:27.032570 1178 raft_consensus.cc:738] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.033061 449 catalog_manager.cc:6596] ReplaceTablet: tablet 953d2ec8a7564c0bbcf0166487dd1746 deleted and replaced by tablet 003d8af0aadb4d68b8eada13b7e94381
I20250112 20:51:27.033370 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_639d154365b14495aee0bffa52ccb7f1" } new_table_name: "default.Table_ab141f89a5bc48c1b0e8d83f0975d74f" modify_external_catalogs: true
I20250112 20:51:27.033499 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 with delete_type TABLET_DATA_DELETED (replaced by tablet 003d8af0aadb4d68b8eada13b7e94381) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.034215 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 with delete_type TABLET_DATA_DELETED (replaced by tablet 003d8af0aadb4d68b8eada13b7e94381) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.035877 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:27.036172 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 with delete_type TABLET_DATA_DELETED (replaced by tablet 003d8af0aadb4d68b8eada13b7e94381) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.036362 447 catalog_manager.cc:6596] ReplaceTablet: tablet ed45eb5cbd2e489c83c3d343ca1adcfc deleted and replaced by tablet e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:27.036856 886 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc with delete_type TABLET_DATA_DELETED (replaced by tablet e72fda1743fc4f96b29add6e36e884de) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.037524 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc with delete_type TABLET_DATA_DELETED (replaced by tablet e72fda1743fc4f96b29add6e36e884de) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.037638 446 catalog_manager.cc:5526] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:27.038257 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:27.038311 1185 ts_tablet_manager.cc:1918] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:27.038450 1185 log.cc:1198] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/eb1dffbe43ba4eeab7ee12d901b62ffb
I20250112 20:51:27.038477 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 4390c60d6c724190a216ef9c961b9592 (table default.Table_a8f2b2f1a79c4ad6a6defba7d45b05c7 [id=2ee1f97fef0f4e2b83932132d0936504]). Expected version 1 got 0
I20250112 20:51:27.038847 1185 ts_tablet_manager.cc:1939] T eb1dffbe43ba4eeab7ee12d901b62ffb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:27.038833 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc with delete_type TABLET_DATA_DELETED (replaced by tablet e72fda1743fc4f96b29add6e36e884de) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.038727 446 catalog_manager.cc:5526] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:27.039125 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:27.039336 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 4d616a8ad9ad4777b44deb77bc321d66 (table default.Table_afbd47897b344f1db9916ab7e703ab36 [id=2fd5583e6e114bcdbdbc723643735fd8]). Expected version 2 got 0
I20250112 20:51:27.039808 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 296703eedc6d459880a74b38e642bceb
I20250112 20:51:27.039853 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet eb1dffbe43ba4eeab7ee12d901b62ffb (table default.Table_20471004d83e485dbea6520c7bdb37b0 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
I20250112 20:51:27.040513 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 296703eedc6d459880a74b38e642bceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.040495 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4ffedc1a9fee40b4bf3a93f178099ceb" } modify_external_catalogs: true
I20250112 20:51:27.040650 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4ffedc1a9fee40b4bf3a93f178099ceb" } modify_external_catalogs: true
I20250112 20:51:27.041378 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 296703eedc6d459880a74b38e642bceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:27.041771 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc63526f932841509277f4726303c801 consensus state has no leader
I20250112 20:51:27.042232 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0bec6474ab5a44ad91c57be6219a3f10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.042802 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 296703eedc6d459880a74b38e642bceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:27.042868 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bec6474ab5a44ad91c57be6219a3f10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.043187 1178 consensus_queue.cc:260] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.045209 1178 ts_tablet_manager.cc:1428] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 20:51:27.045493 1178 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:27.048801 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_65ffc39031ec45e48d7d61a45e2bdf08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.049768 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ffc39031ec45e48d7d61a45e2bdf08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.059350 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_db6e2537abb1488faa40d8119ac5f537"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.060031 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db6e2537abb1488faa40d8119ac5f537 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:27.061270 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1c7a54ce78854137ab2d12e9ce055565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c7a54ce78854137ab2d12e9ce055565 already in progress: creating tablet
I20250112 20:51:27.062166 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 003d8af0aadb4d68b8eada13b7e94381
W20250112 20:51:27.062232 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 304e1c8abe094bd99a99fd358b697f81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 304e1c8abe094bd99a99fd358b697f81 already in progress: creating tablet
I20250112 20:51:27.062567 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ce63edabd93a42389a8c93afb7f4cf00" } new_table_name: "default.Table_39bacb349abc4e21ab442b1e55db7a0e" modify_external_catalogs: true
I20250112 20:51:27.063169 1184 tablet_replica.cc:331] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:27.063267 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7df227e3389841259dc1420ec800e1c1 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
I20250112 20:51:27.063395 1184 raft_consensus.cc:2238] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:27.063673 1184 pending_rounds.cc:62] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:27.063750 1184 pending_rounds.cc:69] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584996185726976 op_type: NO_OP noop_request { }
I20250112 20:51:27.063865 1184 raft_consensus.cc:2883] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:27.063998 1184 raft_consensus.cc:2267] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:27.064016 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3ade9b410d8443249730cacdf7266eb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ade9b410d8443249730cacdf7266eb2 already in progress: creating tablet
I20250112 20:51:27.064841 1184 ts_tablet_manager.cc:1905] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.064980 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7860329544e84de9a97317609ccee5f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.065259 1185 tablet_replica.cc:331] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:27.065467 1185 raft_consensus.cc:2238] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:27.065567 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7860329544e84de9a97317609ccee5f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.065613 1185 raft_consensus.cc:2267] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:27.069767 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c3f359655c94f1d905943069548845b has no consensus state
I20250112 20:51:27.070144 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.071182 1178 tablet_bootstrap.cc:654] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.068892 1185 ts_tablet_manager.cc:1905] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.072449 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:39271: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111),
I20250112 20:51:27.072603 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 20:51:27.073935 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:27.077418 1227 consensus_queue.cc:1035] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.082913 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:27.084841 1178 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:27.085014 1178 ts_tablet_manager.cc:1397] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250112 20:51:27.085249 1218 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39271
I20250112 20:51:27.085282 1410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39271 every 8 connection(s)
I20250112 20:51:27.087019 1218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
I20250112 20:51:27.087574 1178 raft_consensus.cc:357] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.087841 1178 raft_consensus.cc:383] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.087917 1178 raft_consensus.cc:738] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.090451 1178 consensus_queue.cc:260] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.092801 1178 ts_tablet_manager.cc:1428] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:27.093117 1178 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:27.094730 1178 tablet_bootstrap.cc:654] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.096053 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 1218
I20250112 20:51:27.098723 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b32dc02497f64a2e9ad99da6c9115d5d
I20250112 20:51:27.099620 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.100353 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.101270 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.101662 455 catalog_manager.cc:6596] ReplaceTablet: tablet b32dc02497f64a2e9ad99da6c9115d5d deleted and replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:27.105403 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7860329544e84de9a97317609ccee5f8" } new_table_name: "default.Table_342a368a27a14267af730e53365d41d1" modify_external_catalogs: true
W20250112 20:51:27.111006 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f012db68caf4d7d86afa6deb9f2ffaa consensus state has no leader
I20250112 20:51:27.115278 890 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c (DEFAULT_TABLE table=default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.115521 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3f359655c94f1d905943069548845b (DEFAULT_TABLE table=default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.115712 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c0712ab2de43eea8c77ab39a971a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.115903 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3f359655c94f1d905943069548845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.117605 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.118140 749 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c (DEFAULT_TABLE table=default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.118566 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c0712ab2de43eea8c77ab39a971a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.118681 748 tablet_service.cc:1467] Processing CreateTablet for tablet 24c8dc85ac9240598d6e413deac19abe (DEFAULT_TABLE table=default.Table_95ebb558399d410fb0df44b18fdb3603 [id=3a4de1859ffd4db3931ce68febdd8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.118824 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.119026 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.119095 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c8dc85ac9240598d6e413deac19abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.119379 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3f359655c94f1d905943069548845b (DEFAULT_TABLE table=default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.119730 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3f359655c94f1d905943069548845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.124545 752 tablet_service.cc:1467] Processing CreateTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.125077 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.125402 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.128609 1185 ts_tablet_manager.cc:1918] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:27.128832 1185 log.cc:1198] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:27.129055 756 tablet_service.cc:1467] Processing CreateTablet for tablet d770e0399e80491f8de9b08d1eb2f139 (DEFAULT_TABLE table=default.Table_4ff8c0d47d9e4525a7fa3b9304dc109e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.129179 1185 ts_tablet_manager.cc:1939] T 953d2ec8a7564c0bbcf0166487dd1746 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:27.129428 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d770e0399e80491f8de9b08d1eb2f139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.130133 1411 sys_catalog.cc:263] Verifying existing consensus state
I20250112 20:51:27.118681 751 tablet_service.cc:1467] Processing CreateTablet for tablet f769533bfde0448792d01b883e41763c (DEFAULT_TABLE table=default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.133327 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f769533bfde0448792d01b883e41763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.137470 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f769533bfde0448792d01b883e41763c (DEFAULT_TABLE table=default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.137810 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.137917 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f769533bfde0448792d01b883e41763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.138027 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c86ad5f59d431e9b1a1fa890fd35c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.138175 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c86ad5f59d431e9b1a1fa890fd35c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.138427 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 24c8dc85ac9240598d6e413deac19abe (DEFAULT_TABLE table=default.Table_95ebb558399d410fb0df44b18fdb3603 [id=3a4de1859ffd4db3931ce68febdd8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.138492 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c (DEFAULT_TABLE table=default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.138844 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c8dc85ac9240598d6e413deac19abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.139055 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3f359655c94f1d905943069548845b (DEFAULT_TABLE table=default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.139408 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3f359655c94f1d905943069548845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.141134 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f5073a477614ddfb8857091e89bc988"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.141707 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f5073a477614ddfb8857091e89bc988 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.145092 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]). Expected version 1 got 0
I20250112 20:51:27.145130 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.145350 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e0f1c76db0d64624bd43f0e538f0649d (table default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981 [id=5178ded07b844cf4abc1616c387b8585]). Expected version 1 got 0
I20250112 20:51:27.145258 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1bea96e02dff4af98e79dc9e9ab83956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.145521 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.145530 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0871d4272e874a958174f96bfc0e1a95 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]). Expected version 1 got 0
W20250112 20:51:27.145817 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bea96e02dff4af98e79dc9e9ab83956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.147749 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.149053 1028 tablet_service.cc:1467] Processing CreateTablet for tablet d770e0399e80491f8de9b08d1eb2f139 (DEFAULT_TABLE table=default.Table_4ff8c0d47d9e4525a7fa3b9304dc109e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.149398 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d770e0399e80491f8de9b08d1eb2f139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.153204 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c0712ab2de43eea8c77ab39a971a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.117070 893 tablet_service.cc:1467] Processing CreateTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.155876 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.156351 888 tablet_service.cc:1467] Processing CreateTablet for tablet d770e0399e80491f8de9b08d1eb2f139 (DEFAULT_TABLE table=default.Table_4ff8c0d47d9e4525a7fa3b9304dc109e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.159876 891 tablet_service.cc:1467] Processing CreateTablet for tablet 24c8dc85ac9240598d6e413deac19abe (DEFAULT_TABLE table=default.Table_95ebb558399d410fb0df44b18fdb3603 [id=3a4de1859ffd4db3931ce68febdd8836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.160295 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c86ad5f59d431e9b1a1fa890fd35c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.160511 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.161123 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d770e0399e80491f8de9b08d1eb2f139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.161669 886 tablet_service.cc:1467] Processing CreateTablet for tablet f769533bfde0448792d01b883e41763c (DEFAULT_TABLE table=default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.162277 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c8dc85ac9240598d6e413deac19abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.163031 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f769533bfde0448792d01b883e41763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.163635 1411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Bootstrap starting.
I20250112 20:51:27.174081 1308 raft_consensus.cc:491] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.174257 1308 raft_consensus.cc:513] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.174991 1308 leader_election.cc:290] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.175731 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:27.175971 776 raft_consensus.cc:2463] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:27.176510 847 leader_election.cc:304] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:27.176759 1308 raft_consensus.cc:2798] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.176740 1411 log.cc:826] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:27.176880 1308 raft_consensus.cc:491] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.176970 1308 raft_consensus.cc:3054] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.176891 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_71341c6c973c4559949e989d69912a93" } new_table_name: "default.Table_77fce9ea810348bea460274d6f69db14" modify_external_catalogs: true
I20250112 20:51:27.180500 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:27.188361 847 leader_election.cc:343] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.189482 1163 tablet_bootstrap.cc:492] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:27.189623 1163 ts_tablet_manager.cc:1397] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.295s user 0.002s sys 0.020s
I20250112 20:51:27.190419 1163 raft_consensus.cc:357] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.190596 1163 raft_consensus.cc:383] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:27.190558 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23d5269a5c8c4c049265e18bff4e2c45 has no consensus state
I20250112 20:51:27.190685 1163 raft_consensus.cc:738] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.190896 1163 consensus_queue.cc:260] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.191501 1163 ts_tablet_manager.cc:1428] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:27.191809 1163 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:27.197121 1163 tablet_bootstrap.cc:654] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.199076 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:27.199704 1303 consensus_queue.cc:1035] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.201399 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8d5818f34258400982f08f02014c11bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.202968 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d5818f34258400982f08f02014c11bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.207608 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c16f29e9cecc475a82c8f35e9b2e27ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.208114 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c16f29e9cecc475a82c8f35e9b2e27ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.211716 1303 raft_consensus.cc:491] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.211740 1308 raft_consensus.cc:513] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.211906 1303 raft_consensus.cc:513] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.212193 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4c35fe9b46fd41b78ea53fd616abc99b" } modify_external_catalogs: true
I20250112 20:51:27.212399 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4c35fe9b46fd41b78ea53fd616abc99b" } modify_external_catalogs: true
I20250112 20:51:27.212491 1303 leader_election.cc:290] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.212675 1308 leader_election.cc:290] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.214004 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:27.214470 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:27.215011 710 leader_election.cc:343] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.215759 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:27.216028 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:27.216241 776 raft_consensus.cc:3054] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.217150 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f012db68caf4d7d86afa6deb9f2ffaa" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:27.219633 776 raft_consensus.cc:2463] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:27.221969 910 raft_consensus.cc:2463] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:27.222441 710 leader_election.cc:304] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:27.222905 1303 raft_consensus.cc:2798] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.222994 1303 raft_consensus.cc:491] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.223100 1303 raft_consensus.cc:3054] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:27.225059 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8aaa5246382745ecbea4b9bd4825e8f1 consensus state has no leader
I20250112 20:51:27.225843 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 313a5c9905bc445893f1ffa8752ca589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:24 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.226717 1303 raft_consensus.cc:513] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.227346 1303 leader_election.cc:290] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.228063 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:27.228642 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 00c0712ab2de43eea8c77ab39a971a2c
I20250112 20:51:27.229367 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0218ef4723ae4dcd96e64a39634592cf" } modify_external_catalogs: true
I20250112 20:51:27.229570 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0218ef4723ae4dcd96e64a39634592cf" } modify_external_catalogs: true
I20250112 20:51:27.229718 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:27.229959 710 leader_election.cc:343] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.230043 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.230890 847 leader_election.cc:304] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:27.231247 847 leader_election.cc:343] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.231746 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.232014 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:27.232437 910 raft_consensus.cc:3054] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.235394 1186 ts_tablet_manager.cc:1905] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.236022 1354 raft_consensus.cc:491] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.236227 1354 raft_consensus.cc:513] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.237069 1354 leader_election.cc:290] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:27.237887 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:27.238088 776 raft_consensus.cc:2463] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:27.238448 983 leader_election.cc:304] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:27.238716 1354 raft_consensus.cc:2798] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.238791 1354 raft_consensus.cc:491] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.238875 1354 raft_consensus.cc:3054] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.240186 1164 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:27.240332 1164 ts_tablet_manager.cc:1397] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.293s user 0.012s sys 0.019s
I20250112 20:51:27.240293 1422 tablet.cc:1721] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:27.241333 1164 raft_consensus.cc:357] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.241657 1164 raft_consensus.cc:383] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.241744 1164 raft_consensus.cc:738] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.241870 1354 raft_consensus.cc:513] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.243101 1354 leader_election.cc:290] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:27.243443 1164 consensus_queue.cc:260] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.244192 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:27.246299 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cc561e5052ca4268b3733b33a981bf86" } modify_external_catalogs: true
I20250112 20:51:27.246507 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cc561e5052ca4268b3733b33a981bf86" } modify_external_catalogs: true
I20250112 20:51:27.248350 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3a97fd937df444b7a25d357551f5cd85" } new_table_name: "default.Table_96bb34aa833040378700d550b97d143e" modify_external_catalogs: true
I20250112 20:51:27.254832 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ab43d30cd9214becbb707a604ec4b376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.255653 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab43d30cd9214becbb707a604ec4b376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.255709 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:27.256363 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:27.256572 911 raft_consensus.cc:3054] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.254846 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1a635c00d86942f2a0bbd3dfd848b4d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.257939 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.258654 1308 raft_consensus.cc:2798] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.258885 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48df2ec556d43a3b22129b1177ebf74" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:27.264716 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_be025c70ef21488baea67cc8bd9c2d3d" } modify_external_catalogs: true
I20250112 20:51:27.264982 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_be025c70ef21488baea67cc8bd9c2d3d" } modify_external_catalogs: true
I20250112 20:51:27.267421 1184 ts_tablet_manager.cc:1918] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:27.267622 1184 log.cc:1198] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/382aefb4a5e34999b560a9237839b380
I20250112 20:51:27.267597 1308 raft_consensus.cc:695] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:27.267936 776 raft_consensus.cc:3054] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.268878 1164 ts_tablet_manager.cc:1428] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:27.269582 1184 ts_tablet_manager.cc:1939] T 382aefb4a5e34999b560a9237839b380 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:27.272605 1303 raft_consensus.cc:491] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.277222 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_46fa14b9681540c5a4649563f92f7a53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.277724 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e040383f6e2a4ccf9614fd2d119a3cb4
W20250112 20:51:27.278092 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46fa14b9681540c5a4649563f92f7a53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.278458 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.278499 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_22423ed5e6774aa6b41f72eb92b3ab3b" } new_table_name: "default.Table_3916838fde7f4a418d36fc0411fd0908" modify_external_catalogs: true
I20250112 20:51:27.283303 776 raft_consensus.cc:2463] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:27.284009 1185 tablet_replica.cc:331] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:27.292665 1185 raft_consensus.cc:2238] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:27.293066 1185 raft_consensus.cc:2267] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.293897 1185 ts_tablet_manager.cc:1905] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.295511 755 tablet_service.cc:1467] Processing CreateTablet for tablet bc4629339765453d988953799095b3ac (DEFAULT_TABLE table=default.Table_65ffc39031ec45e48d7d61a45e2bdf08 [id=e925fdbaa9964c3aa40462eaa42736cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.295781 888 tablet_service.cc:1467] Processing CreateTablet for tablet bc4629339765453d988953799095b3ac (DEFAULT_TABLE table=default.Table_65ffc39031ec45e48d7d61a45e2bdf08 [id=e925fdbaa9964c3aa40462eaa42736cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.295907 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4629339765453d988953799095b3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.296170 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4629339765453d988953799095b3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.297519 754 tablet_service.cc:1467] Processing CreateTablet for tablet d8330fa1e4654a1a9f167a1120a103fc (DEFAULT_TABLE table=default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.298092 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bd84ab3a8fc4580b971d6b9adb18d6f consensus state has no leader
I20250112 20:51:27.298372 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8330fa1e4654a1a9f167a1120a103fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.298897 891 tablet_service.cc:1467] Processing CreateTablet for tablet ac4e304a1212412f8e99bfd222a110b5 (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.299243 885 tablet_service.cc:1467] Processing CreateTablet for tablet e72fda1743fc4f96b29add6e36e884de (DEFAULT_TABLE table=default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.299268 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4e304a1212412f8e99bfd222a110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.299619 892 tablet_service.cc:1467] Processing CreateTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 (DEFAULT_TABLE table=default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.299960 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201a7e2f17bb46fa918dad8f878a4065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.300228 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72fda1743fc4f96b29add6e36e884de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.302121 749 tablet_service.cc:1467] Processing CreateTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 (DEFAULT_TABLE table=default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.302438 748 tablet_service.cc:1467] Processing CreateTablet for tablet e72fda1743fc4f96b29add6e36e884de (DEFAULT_TABLE table=default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.302520 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201a7e2f17bb46fa918dad8f878a4065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.302763 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72fda1743fc4f96b29add6e36e884de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.303156 752 tablet_service.cc:1467] Processing CreateTablet for tablet ac4e304a1212412f8e99bfd222a110b5 (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.303536 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4e304a1212412f8e99bfd222a110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.308104 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_de4f473fa2b4444798541d51c45e5746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.308351 753 tablet_service.cc:1467] Processing CreateTablet for tablet 3c14102aecb743e2a82660039bd0f038 (DEFAULT_TABLE table=default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.308638 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de4f473fa2b4444798541d51c45e5746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.308681 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14102aecb743e2a82660039bd0f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.309868 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c4712f27518944e0b895370616210edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.309989 747 tablet_service.cc:1467] Processing CreateTablet for tablet 97d32367b39a4cce8371377ec7449586 (DEFAULT_TABLE table=default.Table_43c23610617d4b979805a95633d6c29c [id=7ab67c588df54351a13701edb4b072c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.310437 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4712f27518944e0b895370616210edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.311625 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b74872b3e09745a187c142a867c8cd99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.312100 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b74872b3e09745a187c142a867c8cd99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.313351 886 tablet_service.cc:1467] Processing CreateTablet for tablet d8330fa1e4654a1a9f167a1120a103fc (DEFAULT_TABLE table=default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.313997 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3061d11eaeb14b75a2d4147047f6e0c3" } new_table_name: "default.Table_82e60bb453a74f3dbb30f6a81c8d225c" modify_external_catalogs: true
I20250112 20:51:27.315418 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d32367b39a4cce8371377ec7449586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.323211 750 tablet_service.cc:1467] Processing CreateTablet for tablet 480d49569886408a8ce781a93bb5d10d (DEFAULT_TABLE table=default.Table_db6e2537abb1488faa40d8119ac5f537 [id=d46a20ad8b99492395771d94f5dc60fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.327428 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480d49569886408a8ce781a93bb5d10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.334342 746 tablet_service.cc:1467] Processing CreateTablet for tablet e44422f75a744d078cdf4f0997242e22 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.334914 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44422f75a744d078cdf4f0997242e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.342423 744 tablet_service.cc:1467] Processing CreateTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 (DEFAULT_TABLE table=default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.342990 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d5269a5c8c4c049265e18bff4e2c45. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.345474 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcbbefa7b55041a6a6b50b3c96c45833 consensus state has no leader
I20250112 20:51:27.346262 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.350430 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.345841 741 tablet_service.cc:1467] Processing CreateTablet for tablet af1aac31fd32400b9094f8e2de4fe42c (DEFAULT_TABLE table=default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.360076 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:27.361166 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet af1aac31fd32400b9094f8e2de4fe42c
I20250112 20:51:27.361191 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.362012 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f769533bfde0448792d01b883e41763c
I20250112 20:51:27.370249 745 tablet_service.cc:1467] Processing CreateTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 (DEFAULT_TABLE table=default.Table_9e27bc50d3ea443ab9b522fe653d2269 [id=212d41ddb2d248a0bcf3aa02f7b10bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.361975 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7293510f40804e2aabccf92986489dab (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.372058 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.374761 751 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.373903 743 tablet_service.cc:1467] Processing CreateTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.375727 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af1aac31fd32400b9094f8e2de4fe42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.377506 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7293510f40804e2aabccf92986489dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.378633 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c13126e53b46c0a1e7beafe1b0d295. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.380299 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f0875e271d4aaf9dc5da6aa8113ed0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.381337 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2a8f71233d04ca39e49932308e56492 consensus state has no leader
I20250112 20:51:27.375277 1303 raft_consensus.cc:513] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.384821 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb6e37f75e4aebacb04c4553049940 (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.392560 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb6e37f75e4aebacb04c4553049940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.399893 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_de4f473fa2b4444798541d51c45e5746" } new_table_name: "default.Table_49d04bb236144efe8f33e836bc8d2483" modify_external_catalogs: true
I20250112 20:51:27.396446 749 tablet_service.cc:1467] Processing CreateTablet for tablet b84624dc685b401192ca753ff59da016 (DEFAULT_TABLE table=default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.400722 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84624dc685b401192ca753ff59da016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.401880 753 tablet_service.cc:1467] Processing CreateTablet for tablet 64cd83b199bb4da28d39da449945c389 (DEFAULT_TABLE table=default.Table_8d5818f34258400982f08f02014c11bf [id=370444777ab4411f837c19bc3824376a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.403071 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:27.406127 710 leader_election.cc:343] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.408298 1303 leader_election.cc:290] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.408514 756 tablet_service.cc:1467] Processing CreateTablet for tablet 99ba8529a287434b9ba78e0b6fa7817e (DEFAULT_TABLE table=default.Table_c16f29e9cecc475a82c8f35e9b2e27ed [id=69e87fbf3a29404d92d4b2902996d147]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.408629 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4688dd6bbfc046b7bb140206cae9f104 has no consensus state
I20250112 20:51:27.408881 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ba8529a287434b9ba78e0b6fa7817e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.408918 1308 consensus_queue.cc:237] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.410820 747 tablet_service.cc:1467] Processing CreateTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 (DEFAULT_TABLE table=default.Table_1bea96e02dff4af98e79dc9e9ab83956 [id=7dc6e9f691804365a9696db3b26700bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.413419 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8330fa1e4654a1a9f167a1120a103fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.416059 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e590f9615c4d47abaad1a36ac74b3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.421301 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cd83b199bb4da28d39da449945c389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.431032 746 tablet_service.cc:1467] Processing CreateTablet for tablet f249d70b116443b7823e093efeda5cc3 (DEFAULT_TABLE table=default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.431486 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f249d70b116443b7823e093efeda5cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.431896 1307 raft_consensus.cc:491] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.432060 1307 raft_consensus.cc:513] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.433178 1307 leader_election.cc:290] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.437968 1303 raft_consensus.cc:491] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.441850 883 tablet_service.cc:1467] Processing CreateTablet for tablet e44422f75a744d078cdf4f0997242e22 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.442045 882 tablet_service.cc:1467] Processing CreateTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 (DEFAULT_TABLE table=default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.442471 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44422f75a744d078cdf4f0997242e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.442685 878 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.443239 878 tablet_service.cc:1467] Processing CreateTablet for tablet f249d70b116443b7823e093efeda5cc3 (DEFAULT_TABLE table=default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.443838 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f249d70b116443b7823e093efeda5cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.445020 876 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.445186 889 tablet_service.cc:1467] Processing CreateTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 (DEFAULT_TABLE table=default.Table_9e27bc50d3ea443ab9b522fe653d2269 [id=212d41ddb2d248a0bcf3aa02f7b10bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.446189 877 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.446419 911 raft_consensus.cc:2463] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:27.448890 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c13126e53b46c0a1e7beafe1b0d295. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.449928 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:27.448661 874 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb6e37f75e4aebacb04c4553049940 (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.447937 1303 raft_consensus.cc:513] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.451267 774 raft_consensus.cc:2463] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:27.453203 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ca3d4ace4e204dd799bd05ed211341a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.449388 881 tablet_service.cc:1467] Processing CreateTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.445398 876 tablet_service.cc:1467] Processing CreateTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 (DEFAULT_TABLE table=default.Table_1bea96e02dff4af98e79dc9e9ab83956 [id=7dc6e9f691804365a9696db3b26700bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.444222 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7293510f40804e2aabccf92986489dab (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.453943 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca3d4ace4e204dd799bd05ed211341a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.454632 750 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe28f23448c47d3979f1982f939e39c (DEFAULT_TABLE table=default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.455724 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe28f23448c47d3979f1982f939e39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.455585 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6b82fb5b4ad04365963324b64a52c22b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.456204 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b82fb5b4ad04365963324b64a52c22b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.456954 893 tablet_service.cc:1467] Processing CreateTablet for tablet 97d32367b39a4cce8371377ec7449586 (DEFAULT_TABLE table=default.Table_43c23610617d4b979805a95633d6c29c [id=7ab67c588df54351a13701edb4b072c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.457657 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d5269a5c8c4c049265e18bff4e2c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.457559 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_82e60bb453a74f3dbb30f6a81c8d225c" } new_table_name: "default.Table_d8aa49cfb72d47f889b7646990bdaece" modify_external_catalogs: true
I20250112 20:51:27.457396 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 (DEFAULT_TABLE table=default.Table_49d04bb236144efe8f33e836bc8d2483 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.449939 877 tablet_service.cc:1467] Processing CreateTablet for tablet 64cd83b199bb4da28d39da449945c389 (DEFAULT_TABLE table=default.Table_8d5818f34258400982f08f02014c11bf [id=370444777ab4411f837c19bc3824376a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.458968 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e3b1f099596246f78012a56ae87a63bd" } modify_external_catalogs: true
I20250112 20:51:27.459188 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e3b1f099596246f78012a56ae87a63bd" } modify_external_catalogs: true
I20250112 20:51:27.456836 847 leader_election.cc:304] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:27.458583 879 tablet_service.cc:1467] Processing CreateTablet for tablet af1aac31fd32400b9094f8e2de4fe42c (DEFAULT_TABLE table=default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.462332 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3c14102aecb743e2a82660039bd0f038 (DEFAULT_TABLE table=default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.468518 890 tablet_service.cc:1467] Processing CreateTablet for tablet 480d49569886408a8ce781a93bb5d10d (DEFAULT_TABLE table=default.Table_db6e2537abb1488faa40d8119ac5f537 [id=d46a20ad8b99492395771d94f5dc60fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.469353 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f0875e271d4aaf9dc5da6aa8113ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.471324 875 tablet_service.cc:1467] Processing CreateTablet for tablet b84624dc685b401192ca753ff59da016 (DEFAULT_TABLE table=default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.474344 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8f636df32bd34438bb54989736eaa431 (DEFAULT_TABLE table=default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.475631 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f636df32bd34438bb54989736eaa431. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.477257 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2a8f71233d04ca39e49932308e56492 consensus state has no leader
I20250112 20:51:27.478984 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_941f7aeb51a54bf38161efd28eb35340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.479322 739 tablet_service.cc:1467] Processing CreateTablet for tablet 71743c73b6f24da9b14a3d4148117813 (DEFAULT_TABLE table=default.Table_b74872b3e09745a187c142a867c8cd99 [id=a463efa09ff246029307751a49e80170]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.479635 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_941f7aeb51a54bf38161efd28eb35340 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.480726 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71743c73b6f24da9b14a3d4148117813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.481117 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c4712f27518944e0b895370616210edb" } modify_external_catalogs: true
I20250112 20:51:27.481320 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c4712f27518944e0b895370616210edb" } modify_external_catalogs: true
I20250112 20:51:27.481432 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480d49569886408a8ce781a93bb5d10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.484491 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2783ab15d9c84d9a9b09c3688bce6109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.485186 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2783ab15d9c84d9a9b09c3688bce6109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.497038 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:27.498281 1307 raft_consensus.cc:2798] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.498411 1307 raft_consensus.cc:491] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.498522 1307 raft_consensus.cc:3054] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.501400 1307 raft_consensus.cc:513] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.501664 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cd83b199bb4da28d39da449945c389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.503300 1303 leader_election.cc:290] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.503850 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.505139 888 tablet_service.cc:1467] Processing CreateTablet for tablet 99ba8529a287434b9ba78e0b6fa7817e (DEFAULT_TABLE table=default.Table_c16f29e9cecc475a82c8f35e9b2e27ed [id=69e87fbf3a29404d92d4b2902996d147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.510321 910 raft_consensus.cc:2463] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:27.511191 710 leader_election.cc:304] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:27.511334 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0871d4272e874a958174f96bfc0e1a95" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:27.511191 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4688dd6bbfc046b7bb140206cae9f104. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.511863 710 leader_election.cc:343] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.512028 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af1aac31fd32400b9094f8e2de4fe42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.512975 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3916838fde7f4a418d36fc0411fd0908" } new_table_name: "default.Table_69153618434846ad9d90837893bea2da" modify_external_catalogs: true
I20250112 20:51:27.514854 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a00d407cb3be4a92a953402f4590840a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.515573 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a00d407cb3be4a92a953402f4590840a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.516290 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84624dc685b401192ca753ff59da016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.517654 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cad323ed70af4a0a91136d819296a4da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.518225 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cad323ed70af4a0a91136d819296a4da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.524879 1307 leader_election.cc:290] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.532063 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7293510f40804e2aabccf92986489dab. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.537096 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcbbefa7b55041a6a6b50b3c96c45833 consensus state has no leader
I20250112 20:51:27.540360 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb6e37f75e4aebacb04c4553049940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.541217 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d32367b39a4cce8371377ec7449586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.543030 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e590f9615c4d47abaad1a36ac74b3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.531212 1430 consensus_queue.cc:1035] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20250112 20:51:27.550246 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4ff8c0d47d9e4525a7fa3b9304dc109e" } new_table_name: "default.Table_22041bd544d842dd8bba1f206c270a4e" modify_external_catalogs: true
I20250112 20:51:27.552024 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_18e26eeea5ff4c4196d49055411fb3ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:27.552320 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14102aecb743e2a82660039bd0f038. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.552606 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e26eeea5ff4c4196d49055411fb3ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:27.553045 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 80496.3us
I20250112 20:51:27.555738 1427 raft_consensus.cc:491] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.555876 1427 raft_consensus.cc:513] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.556649 1427 leader_election.cc:290] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.556849 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ba8529a287434b9ba78e0b6fa7817e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.557314 1427 raft_consensus.cc:2798] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.557458 1427 raft_consensus.cc:695] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:27.557744 1427 consensus_queue.cc:237] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:27.561520 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d770e0399e80491f8de9b08d1eb2f139 consensus state has no leader
I20250112 20:51:27.567849 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc82df6f9ef0472db33f87ad8da54503" } modify_external_catalogs: true
I20250112 20:51:27.568029 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc82df6f9ef0472db33f87ad8da54503" } modify_external_catalogs: true
I20250112 20:51:27.569697 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ac6a910be76848cf9870b52ef755a7b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.570266 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac6a910be76848cf9870b52ef755a7b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.571595 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_30306a177be64cc2b4c2a1ed3351b4de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.572185 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30306a177be64cc2b4c2a1ed3351b4de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.574097 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ab43d30cd9214becbb707a604ec4b376" } modify_external_catalogs: true
I20250112 20:51:27.574322 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ab43d30cd9214becbb707a604ec4b376" } modify_external_catalogs: true
I20250112 20:51:27.575774 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f49b62d82ca74b20896a1eac33adf71a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.576354 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f49b62d82ca74b20896a1eac33adf71a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.578773 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_80526018642b4dd5882083cb240f243d" } modify_external_catalogs: true
I20250112 20:51:27.578994 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_80526018642b4dd5882083cb240f243d" } modify_external_catalogs: true
I20250112 20:51:27.582402 446 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_088900cac1084a659ca9af91b6c0276b [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 1 got 0
I20250112 20:51:27.582942 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616b8aa8fd62486c978c275712064f77" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:27.584383 710 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.587011 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe28f23448c47d3979f1982f939e39c (DEFAULT_TABLE table=default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.587044 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a00d407cb3be4a92a953402f4590840a" } new_table_name: "default.Table_558ea8a4fda94f119e0603dcb4ceff52" modify_external_catalogs: true
I20250112 20:51:27.595844 877 tablet_service.cc:1467] Processing CreateTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 (DEFAULT_TABLE table=default.Table_49d04bb236144efe8f33e836bc8d2483 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.596691 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0e2235e2a63c4e1192b3260085908445
I20250112 20:51:27.596997 881 tablet_service.cc:1467] Processing CreateTablet for tablet 8f636df32bd34438bb54989736eaa431 (DEFAULT_TABLE table=default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.597146 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe28f23448c47d3979f1982f939e39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.598618 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:27.598845 776 raft_consensus.cc:3054] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.602573 776 raft_consensus.cc:2463] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:27.603262 1432 tablet.cc:1721] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:27.603971 1303 consensus_queue.cc:1035] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.605545 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.608736 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4688dd6bbfc046b7bb140206cae9f104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.611052 1186 ts_tablet_manager.cc:1918] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:27.619324 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.623193 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f636df32bd34438bb54989736eaa431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.627586 1186 log.cc:1198] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/aa3bd4eb92114bdb98b923fda9192cd6
I20250112 20:51:27.628716 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b84624dc685b401192ca753ff59da016
I20250112 20:51:27.630796 1186 ts_tablet_manager.cc:1939] T aa3bd4eb92114bdb98b923fda9192cd6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:27.641808 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8b414be57990432bb786daf5faf0cc2e
I20250112 20:51:27.642791 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 121) took 1046 ms. Trace:
I20250112 20:51:27.644618 1432 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:27.650635 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d8330fa1e4654a1a9f167a1120a103fc
I20250112 20:51:27.647740 1186 rpcz_store.cc:276] 0112 20:51:26.585085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.585148 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:26.998391 (+413243us) ts_tablet_manager.cc:1145] Deleting tablet aa3bd4eb92114bdb98b923fda9192cd6
0112 20:51:26.998402 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:27.474320 (+475918us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:27.610935 (+136615us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:27.631107 (+ 20172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":413064}
I20250112 20:51:27.660624 1186 tablet_replica.cc:331] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:27.661417 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eda44c48bf9746b59ccdc4d39a41ea91 has no consensus state
I20250112 20:51:27.665235 879 tablet_service.cc:1467] Processing CreateTablet for tablet 71743c73b6f24da9b14a3d4148117813 (DEFAULT_TABLE table=default.Table_b74872b3e09745a187c142a867c8cd99 [id=a463efa09ff246029307751a49e80170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.666491 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71743c73b6f24da9b14a3d4148117813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.675107 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.676337 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.678006 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:27.680819 748 tablet_service.cc:1467] Processing CreateTablet for tablet 01367a11dea04b33a9255508501b9060 (DEFAULT_TABLE table=default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.681349 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01367a11dea04b33a9255508501b9060. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.705227 1178 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.016s sys 0.024s
W20250112 20:51:27.705365 1178 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.576s user 0.016s sys 0.024s
I20250112 20:51:27.712972 1178 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:27.713251 1178 ts_tablet_manager.cc:1397] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.620s user 0.017s sys 0.026s
I20250112 20:51:27.714232 1178 raft_consensus.cc:357] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.715327 1178 raft_consensus.cc:383] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.715863 1178 raft_consensus.cc:738] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.716606 1178 consensus_queue.cc:260] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.719919 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
W20250112 20:51:27.729554 1272 kernel_stack_watchdog.cc:198] Thread 1411 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 381ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ff3bd4fd980 (unknown) at ??:0
@ 0x926cb7 __sanitizer::internal_sched_yield() at ??:?
@ 0x88c36d __interceptor_pthread_create at ??:?
@ 0x7ff3ba10ddb1 kudu::Thread::StartThread() at ??:0
@ 0xa5e837 kudu::Thread::Create() at ??:?
@ 0x7ff3ba12e7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7ff3ba12a3bd kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7ff3ba12f4ff kudu::ThreadPool::Submit() at ??:0
@ 0x7ff3c5309bcd kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7ff3c5308bbf kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7ff3c5308790 kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7ff3c53106a5 kudu::log::Log::Init() at ??:0
@ 0x7ff3c530fa01 kudu::log::Log::Open() at ??:0
@ 0x7ff3c6c1fe21 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7ff3c6c20935 kudu::tablet::TabletBootstrap::PlaySegments() at ??:0
@ 0x7ff3c6c1c52c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
I20250112 20:51:27.740211 1164 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:27.742601 1164 tablet_bootstrap.cc:654] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:27.745154 1227 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.001s sys 0.014s
W20250112 20:51:27.745270 1227 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.588s user 0.001s sys 0.014s
I20250112 20:51:27.317798 1022 tablet_service.cc:1467] Processing CreateTablet for tablet e72fda1743fc4f96b29add6e36e884de (DEFAULT_TABLE table=default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.746546 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72fda1743fc4f96b29add6e36e884de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.317992 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3c14102aecb743e2a82660039bd0f038 (DEFAULT_TABLE table=default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.749213 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14102aecb743e2a82660039bd0f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.318316 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 480d49569886408a8ce781a93bb5d10d (DEFAULT_TABLE table=default.Table_db6e2537abb1488faa40d8119ac5f537 [id=d46a20ad8b99492395771d94f5dc60fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.753110 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480d49569886408a8ce781a93bb5d10d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.755858 971 kernel_stack_watchdog.cc:198] Thread 983 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 186ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f517acbf980 (unknown) at ??:0
@ 0x7f517acb99c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f5174b73904 google::LogMessage::Flush() at ??:0
@ 0x7f5174b73a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f51830bb2eb kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f51830bc692 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f51830bf05a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f51830bee1e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7f517e580cdf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f517e5808be std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7f517e56c930 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f517e56ce7c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f517e5094ea kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f517e507ea6 kudu::rpc::Connection::ReadHandler() at ??:0
I20250112 20:51:27.316287 983 leader_election.cc:304] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:27.756366 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 472429us
I20250112 20:51:27.756413 1354 raft_consensus.cc:2798] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.320143 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 97d32367b39a4cce8371377ec7449586 (DEFAULT_TABLE table=default.Table_43c23610617d4b979805a95633d6c29c [id=7ab67c588df54351a13701edb4b072c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.319846 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 (DEFAULT_TABLE table=default.Table_9e27bc50d3ea443ab9b522fe653d2269 [id=212d41ddb2d248a0bcf3aa02f7b10bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.756821 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d32367b39a4cce8371377ec7449586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.757027 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c13126e53b46c0a1e7beafe1b0d295. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.320474 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e44422f75a744d078cdf4f0997242e22 (DEFAULT_TABLE table=default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.320824 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 (DEFAULT_TABLE table=default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.757318 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44422f75a744d078cdf4f0997242e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.321702 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 (DEFAULT_TABLE table=default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.757548 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d5269a5c8c4c049265e18bff4e2c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.320766 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (DEFAULT_TABLE table=default.Table_d83a3643f7974d1ab7d542bdc1dea5ea [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.757793 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201a7e2f17bb46fa918dad8f878a4065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.758198 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f0875e271d4aaf9dc5da6aa8113ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.321739 1015 tablet_service.cc:1467] Processing CreateTablet for tablet af1aac31fd32400b9094f8e2de4fe42c (DEFAULT_TABLE table=default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.322163 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 7293510f40804e2aabccf92986489dab (DEFAULT_TABLE table=default.Table_e2f2bcbfc0194e0e9342fe49a37abec0 [id=86e79855b8c7432b8b460e4c16f3be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.322140 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d8330fa1e4654a1a9f167a1120a103fc (DEFAULT_TABLE table=default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.322728 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ac4e304a1212412f8e99bfd222a110b5 (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.323437 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet e040383f6e2a4ccf9614fd2d119a3cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.330036 1163 tablet_bootstrap.cc:492] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:27.364109 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.364259 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.368717 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.381176 1011 tablet_service.cc:1467] Processing CreateTablet for tablet b84624dc685b401192ca753ff59da016 (DEFAULT_TABLE table=default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.383639 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb6e37f75e4aebacb04c4553049940 (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.403856 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.447888 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.503437 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0871d4272e874a958174f96bfc0e1a95" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.558238 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616b8aa8fd62486c978c275712064f77" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.573381 1422 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:27.598646 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48b17628cf145b69e5fdde4131d9fd2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:27.759570 710 leader_election.cc:343] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.604223 1044 raft_consensus.cc:3054] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.759714 710 leader_election.cc:304] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:27.323199 1021 tablet_service.cc:1467] Processing CreateTablet for tablet bc4629339765453d988953799095b3ac (DEFAULT_TABLE table=default.Table_65ffc39031ec45e48d7d61a45e2bdf08 [id=e925fdbaa9964c3aa40462eaa42736cb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:27.760555 710 leader_election.cc:343] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.760601 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af1aac31fd32400b9094f8e2de4fe42c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.760701 710 leader_election.cc:304] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:27.761158 710 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.761265 710 leader_election.cc:304] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:27.761530 1303 raft_consensus.cc:2743] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:27.761713 1303 raft_consensus.cc:2743] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:27.761852 1303 raft_consensus.cc:2743] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:27.769313 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb6e37f75e4aebacb04c4553049940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.773124 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4e304a1212412f8e99bfd222a110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.773581 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 99ba8529a287434b9ba78e0b6fa7817e (DEFAULT_TABLE table=default.Table_c16f29e9cecc475a82c8f35e9b2e27ed [id=69e87fbf3a29404d92d4b2902996d147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.774065 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ba8529a287434b9ba78e0b6fa7817e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.773125 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 64cd83b199bb4da28d39da449945c389 (DEFAULT_TABLE table=default.Table_8d5818f34258400982f08f02014c11bf [id=370444777ab4411f837c19bc3824376a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.777232 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84624dc685b401192ca753ff59da016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.777485 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cd83b199bb4da28d39da449945c389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.777776 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4629339765453d988953799095b3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.779976 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7293510f40804e2aabccf92986489dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.780984 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 121) took 1143 ms. Trace:
I20250112 20:51:27.781102 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8330fa1e4654a1a9f167a1120a103fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:27.782928 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d770e0399e80491f8de9b08d1eb2f139 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d770e0399e80491f8de9b08d1eb2f139 consensus state has no leader
I20250112 20:51:27.782948 748 tablet_service.cc:1467] Processing CreateTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 (DEFAULT_TABLE table=default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.783401 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda44c48bf9746b59ccdc4d39a41ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.783433 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe28f23448c47d3979f1982f939e39c (DEFAULT_TABLE table=default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.784296 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 (DEFAULT_TABLE table=default.Table_49d04bb236144efe8f33e836bc8d2483 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.786250 1163 ts_tablet_manager.cc:1397] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.590s user 0.007s sys 0.006s
I20250112 20:51:27.787977 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 (DEFAULT_TABLE table=default.Table_1bea96e02dff4af98e79dc9e9ab83956 [id=7dc6e9f691804365a9696db3b26700bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.788441 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe28f23448c47d3979f1982f939e39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.790197 1163 raft_consensus.cc:357] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.790685 1163 raft_consensus.cc:383] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.790766 1163 raft_consensus.cc:738] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.791354 1163 consensus_queue.cc:260] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.792089 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e590f9615c4d47abaad1a36ac74b3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.792629 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4688dd6bbfc046b7bb140206cae9f104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.805810 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9dfb4678229745ae83688748826dc594" } new_table_name: "default.Table_05c8c0dc76884c47b0e87e2801dffe72" modify_external_catalogs: true
I20250112 20:51:27.806079 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_088900cac1084a659ca9af91b6c0276b" } new_table_name: "default.Table_673a2a9243984736b6b43018eb3e1a42" modify_external_catalogs: true
I20250112 20:51:27.806376 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ff1653ec182d4a2bbdd3c2a01b91d268"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.806995 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.807296 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_95ebb558399d410fb0df44b18fdb3603" } new_table_name: "default.Table_2dc9d59ea355480789bf06df8c358e57" modify_external_catalogs: true
I20250112 20:51:27.808522 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a476af27be9d446dab54b21d797ca38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.809113 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a476af27be9d446dab54b21d797ca38f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.810431 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_43f6a278ed0b4388aceea8a274566af9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.810984 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f6a278ed0b4388aceea8a274566af9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.812259 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0fa5226c1b51464da6a1b54f22ef816e" } modify_external_catalogs: true
I20250112 20:51:27.812434 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0fa5226c1b51464da6a1b54f22ef816e" } modify_external_catalogs: true
I20250112 20:51:27.813884 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3c1abf71e9154b45ba412645d3f22386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.814429 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c1abf71e9154b45ba412645d3f22386 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.815752 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_326496344cf44e9d949ee3545f916c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.816322 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_326496344cf44e9d949ee3545f916c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.817637 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:27.818207 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:27.823406 1185 ts_tablet_manager.cc:1918] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:27.823552 1185 log.cc:1198] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:27.823891 1185 ts_tablet_manager.cc:1939] T ed45eb5cbd2e489c83c3d343ca1adcfc P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:27.824434 1185 tablet_replica.cc:331] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:27.824633 1185 raft_consensus.cc:2238] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:27.824831 1185 raft_consensus.cc:2267] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.825376 1185 ts_tablet_manager.cc:1905] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.831202 1428 raft_consensus.cc:491] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.831387 1428 raft_consensus.cc:513] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.831667 1185 ts_tablet_manager.cc:1918] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:27.831853 1185 log.cc:1198] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/296703eedc6d459880a74b38e642bceb
I20250112 20:51:27.832235 1185 ts_tablet_manager.cc:1939] T 296703eedc6d459880a74b38e642bceb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:27.832545 1428 leader_election.cc:290] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:27.835984 1185 tablet_replica.cc:331] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:27.836177 1185 raft_consensus.cc:2238] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:27.836319 1185 raft_consensus.cc:2267] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:27.837139 1411 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.131s sys 0.215s
W20250112 20:51:27.837328 1411 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.660s user 0.132s sys 0.215s
I20250112 20:51:27.837689 1185 ts_tablet_manager.cc:1905] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.783068 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f249d70b116443b7823e093efeda5cc3 (DEFAULT_TABLE table=default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.840210 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f249d70b116443b7823e093efeda5cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.841004 1439 raft_consensus.cc:491] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.841202 1439 raft_consensus.cc:513] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.842340 1439 leader_election.cc:290] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:27.842715 1354 raft_consensus.cc:695] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:27.843067 1354 consensus_queue.cc:237] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:27.846942 750 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.847097 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ed45eb5cbd2e489c83c3d343ca1adcfc (table default.Table_07853a7e990447d293c9fd0530206abf [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:27.847354 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ad7cf1e0c74265a4d21931279f2318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.847865 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 296703eedc6d459880a74b38e642bceb (table default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]) successfully deleted
W20250112 20:51:27.848112 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b32dc02497f64a2e9ad99da6c9115d5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32dc02497f64a2e9ad99da6c9115d5d already in progress: creating tablet
I20250112 20:51:27.848954 1178 ts_tablet_manager.cc:1428] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.136s user 0.003s sys 0.000s
W20250112 20:51:27.850081 913 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.017s sys 0.020s
W20250112 20:51:27.850226 913 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.579s user 0.017s sys 0.020s
I20250112 20:51:27.850353 746 tablet_service.cc:1467] Processing CreateTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 (DEFAULT_TABLE table=default.Table_ca3d4ace4e204dd799bd05ed211341a0 [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.850723 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0daaeb0bf6046da8de969b1f4c71c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.853109 752 tablet_service.cc:1467] Processing CreateTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee (DEFAULT_TABLE table=default.Table_6b82fb5b4ad04365963324b64a52c22b [id=dd64a2679c0849279e9dae49f268056b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.853442 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eb0f79df5444f8a8360921d5f2b9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.853835 1178 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:27.855491 1178 tablet_bootstrap.cc:654] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:27.856333 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24c8dc85ac9240598d6e413deac19abe consensus state has no leader
I20250112 20:51:27.856915 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8f636df32bd34438bb54989736eaa431 (DEFAULT_TABLE table=default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.857097 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 (DEFAULT_TABLE table=default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.718189 1186 raft_consensus.cc:2238] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:27.857426 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet decdec0d34344ec4b422643b0d4e7659 consensus state has no leader
I20250112 20:51:27.857528 1186 pending_rounds.cc:62] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:27.857592 1186 pending_rounds.cc:69] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584994770051072 op_type: NO_OP noop_request { }
I20250112 20:51:27.858258 1186 raft_consensus.cc:2883] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:27.858381 1186 raft_consensus.cc:2267] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.858927 1186 ts_tablet_manager.cc:1905] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.857365 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f636df32bd34438bb54989736eaa431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.803275 847 leader_election.cc:304] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:27.862300 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 59070.1us
I20250112 20:51:27.803813 1308 consensus_queue.cc:1035] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.865357 1308 raft_consensus.cc:2798] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.865520 1308 raft_consensus.cc:695] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:27.865831 1308 consensus_queue.cc:237] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.866933 874 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.867168 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.867326 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:27.867710 881 tablet_service.cc:1467] Processing CreateTablet for tablet 01367a11dea04b33a9255508501b9060 (DEFAULT_TABLE table=default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.868083 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01367a11dea04b33a9255508501b9060. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.869047 877 tablet_service.cc:1467] Processing CreateTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 (DEFAULT_TABLE table=default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.869386 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda44c48bf9746b59ccdc4d39a41ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.873598 887 tablet_service.cc:1467] Processing CreateTablet for tablet d66c55c29de7458bb8ea69b826c567d7 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.874024 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c55c29de7458bb8ea69b826c567d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.877488 876 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.877887 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ad7cf1e0c74265a4d21931279f2318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.881074 890 tablet_service.cc:1467] Processing CreateTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 (DEFAULT_TABLE table=default.Table_ca3d4ace4e204dd799bd05ed211341a0 [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.881470 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0daaeb0bf6046da8de969b1f4c71c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.885111 880 tablet_service.cc:1467] Processing CreateTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee (DEFAULT_TABLE table=default.Table_6b82fb5b4ad04365963324b64a52c22b [id=dd64a2679c0849279e9dae49f268056b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.885483 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eb0f79df5444f8a8360921d5f2b9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.886296 1411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=26 ignored=0} inserts{seen=23 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 20:51:27.887315 1411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Bootstrap complete.
I20250112 20:51:27.889053 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 (DEFAULT_TABLE table=default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.889037 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1c76db0d64624bd43f0e538f0649d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.889348 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8250621cbe45eba669b5ef14e672c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.893761 878 tablet_service.cc:1467] Processing CreateTablet for tablet f4cae067df1a40098f81cfbd17224c75 (DEFAULT_TABLE table=default.Table_ac6a910be76848cf9870b52ef755a7b9 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.894191 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cae067df1a40098f81cfbd17224c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.781157 1184 rpcz_store.cc:276] 0112 20:51:26.637169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.638588 (+ 1419us) service_pool.cc:225] Handling call
0112 20:51:27.063092 (+424504us) ts_tablet_manager.cc:1145] Deleting tablet 382aefb4a5e34999b560a9237839b380
0112 20:51:27.063103 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:27.225860 (+162757us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:27.267295 (+ 41435us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:27.780943 (+513648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":424327}
I20250112 20:51:27.895977 1184 tablet_replica.cc:331] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:27.897166 882 tablet_service.cc:1467] Processing CreateTablet for tablet 54de54eabe5946058d1940b79f5fb832 (DEFAULT_TABLE table=default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.897547 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54de54eabe5946058d1940b79f5fb832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.900386 889 tablet_service.cc:1467] Processing CreateTablet for tablet 572172a274494e429b7e982815897e61 (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.900806 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572172a274494e429b7e982815897e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.834394 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f1c76db0d64624bd43f0e538f0649d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:27.904479 1430 raft_consensus.cc:491] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.904680 1430 raft_consensus.cc:513] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.908214 1430 leader_election.cc:290] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:27.916833 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:27.917558 1163 ts_tablet_manager.cc:1428] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.131s user 0.004s sys 0.002s
I20250112 20:51:27.920187 1184 raft_consensus.cc:2238] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:27.920151 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 71743c73b6f24da9b14a3d4148117813 (DEFAULT_TABLE table=default.Table_b74872b3e09745a187c142a867c8cd99 [id=a463efa09ff246029307751a49e80170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.920367 1184 raft_consensus.cc:2267] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:27.920562 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71743c73b6f24da9b14a3d4148117813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.921602 1184 ts_tablet_manager.cc:1905] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:27.922014 1307 raft_consensus.cc:491] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.922194 1307 raft_consensus.cc:513] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.922955 1307 leader_election.cc:290] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:27.924145 847 leader_election.cc:343] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:27.924515 847 leader_election.cc:343] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:27.929553 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.929765 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.930294 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.930486 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.931864 1163 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:27.926414 1411 raft_consensus.cc:357] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:27.934257 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 01367a11dea04b33a9255508501b9060 (DEFAULT_TABLE table=default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.937687 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:27.938038 1021 tablet_service.cc:1467] Processing CreateTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.939667 1029 tablet_service.cc:1467] Processing CreateTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 (DEFAULT_TABLE table=default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.939952 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01367a11dea04b33a9255508501b9060. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.945607 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d66c55c29de7458bb8ea69b826c567d7 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.946732 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee (DEFAULT_TABLE table=default.Table_6b82fb5b4ad04365963324b64a52c22b [id=dd64a2679c0849279e9dae49f268056b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.947270 1015 tablet_service.cc:1467] Processing CreateTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 (DEFAULT_TABLE table=default.Table_ca3d4ace4e204dd799bd05ed211341a0 [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.952915 753 tablet_service.cc:1467] Processing CreateTablet for tablet 54de54eabe5946058d1940b79f5fb832 (DEFAULT_TABLE table=default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.953302 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54de54eabe5946058d1940b79f5fb832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.957079 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f4cae067df1a40098f81cfbd17224c75 (DEFAULT_TABLE table=default.Table_ac6a910be76848cf9870b52ef755a7b9 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.959050 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 (DEFAULT_TABLE table=default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.960588 1411 raft_consensus.cc:738] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b985fd656ce44131bc91ccfeb79587c5, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.960980 1441 raft_consensus.cc:491] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.961957 1430 consensus_queue.cc:1035] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.964421 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ad7cf1e0c74265a4d21931279f2318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.965951 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda44c48bf9746b59ccdc4d39a41ea91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.966154 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eb0f79df5444f8a8360921d5f2b9ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.966733 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0daaeb0bf6046da8de969b1f4c71c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.967294 1411 consensus_queue.cc:260] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:27.968258 1163 tablet_bootstrap.cc:654] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.969085 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cae067df1a40098f81cfbd17224c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.969661 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8250621cbe45eba669b5ef14e672c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.971310 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 54de54eabe5946058d1940b79f5fb832 (DEFAULT_TABLE table=default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:27.961143 1441 raft_consensus.cc:513] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:27.972498 1441 leader_election.cc:290] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:27.978485 1178 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:28.006762 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 382aefb4a5e34999b560a9237839b380 (table default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]) successfully deleted
I20250112 20:51:27.978649 1178 ts_tablet_manager.cc:1397] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.021s
I20250112 20:51:28.042621 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:28.059676 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b74872b3e09745a187c142a867c8cd99" } new_table_name: "default.Table_d8d06d1114e0487d8f3097053759ceb8" modify_external_catalogs: true
W20250112 20:51:28.302063 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 295288us
I20250112 20:51:28.060998 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f594de5fea864b98a7b2838bf0257f3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.302793 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f594de5fea864b98a7b2838bf0257f3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.302836 1178 raft_consensus.cc:357] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.303092 1178 raft_consensus.cc:383] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.303172 1178 raft_consensus.cc:738] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.303447 1178 consensus_queue.cc:260] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.303952 1178 ts_tablet_manager.cc:1428] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:27.979607 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 572172a274494e429b7e982815897e61 (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.024622 1411 sys_catalog.cc:564] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:27.992208 755 tablet_service.cc:1467] Processing CreateTablet for tablet 572172a274494e429b7e982815897e61 (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.303484 454 catalog_manager.cc:5526] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:28.398767 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:28.008558 1451 sys_catalog.cc:455] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:27.993821 756 tablet_service.cc:1467] Processing CreateTablet for tablet f4cae067df1a40098f81cfbd17224c75 (DEFAULT_TABLE table=default.Table_ac6a910be76848cf9870b52ef755a7b9 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.223268 1357 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58982 (request call id 0) took 1057 ms. Trace:
I20250112 20:51:27.987776 1186 ts_tablet_manager.cc:1918] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:28.399892 1186 log.cc:1198] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8c7a94b2e9be4b5db3dfd6129b03cd25
I20250112 20:51:28.400488 1186 ts_tablet_manager.cc:1939] T 8c7a94b2e9be4b5db3dfd6129b03cd25 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:27.982781 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:27.982942 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c55c29de7458bb8ea69b826c567d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.095089 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cc971026f6b44db6af8215890ba71075
I20250112 20:51:28.402063 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981" } modify_external_catalogs: true
I20250112 20:51:28.402336 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981" } modify_external_catalogs: true
I20250112 20:51:27.994365 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8250621cbe45eba669b5ef14e672c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.986097 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9aa4f42f89455cba7c72e7a49c6a94" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:28.114316 1163 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:28.403985 1163 ts_tablet_manager.cc:1397] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.486s user 0.005s sys 0.000s
W20250112 20:51:28.404285 847 leader_election.cc:343] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:28.404503 847 leader_election.cc:343] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:28.113888 983 leader_election.cc:343] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 719b906973a8490898d6befce49910d3
W20250112 20:51:28.404884 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 292701us
I20250112 20:51:28.404801 1163 raft_consensus.cc:357] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.404989 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54de54eabe5946058d1940b79f5fb832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.405027 1163 raft_consensus.cc:383] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.405129 1163 raft_consensus.cc:738] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.405342 1163 consensus_queue.cc:260] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.405898 1163 ts_tablet_manager.cc:1428] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:28.406184 1163 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:28.407953 1163 tablet_bootstrap.cc:654] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.223543 1454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ae76622ac46f5a584a582e372bc05. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:28.408885 1454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:28.001849 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:28.287420 428 kernel_stack_watchdog.cc:198] Thread 440 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 211ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa2e1409980 (unknown) at ??:0
@ 0x7fa2da15d255 __write at ??:0
@ 0x7fa2da0d828d _IO_file_write at ??:0
@ 0x7fa2da0d8bff _IO_file_xsputn at ??:0
@ 0x7fa2da0cca57 fwrite at ??:0
@ 0x891127 __interceptor_fwrite.part.56 at ??:?
@ 0x7fa2db2c18a0 google::LogMessage::SendToLog() at ??:0
@ 0x7fa2db2bd7cc google::LogMessage::Flush() at ??:0
@ 0x7fa2db2bda11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fa2f13eb978 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7fa2f1344139 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fa2f13ef5c0 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fa2f13ef3de _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
@ 0x7fa2e4ccacdf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fa2e4cca8be std::_Function_handler<>::_M_invoke() at ??:0
I20250112 20:51:28.409308 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572172a274494e429b7e982815897e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.997824 749 tablet_service.cc:1467] Processing CreateTablet for tablet d66c55c29de7458bb8ea69b826c567d7 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.094681 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1f5073a477614ddfb8857091e89bc988" } modify_external_catalogs: true
I20250112 20:51:28.409621 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c55c29de7458bb8ea69b826c567d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.409551 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1f5073a477614ddfb8857091e89bc988" } modify_external_catalogs: true
I20250112 20:51:28.411665 1440 consensus_queue.cc:1035] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.085546 1435 mvcc.cc:204] Tried to move back new op lower bound from 7113584998650155008 to 7113584993868369920. Current Snapshot: MvccSnapshot[applied={T|T < 7113584998650155008}]
I20250112 20:51:28.412945 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572172a274494e429b7e982815897e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.307286 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet aa3bd4eb92114bdb98b923fda9192cd6 (table default.Table_954436533cfc462dbffed1710e7920ca [id=c1536ab3b947496baa4ada09d64c716d]) successfully deleted
I20250112 20:51:28.305640 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4b1f4ea776424da59c8ec69715c5337d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.413375 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 106102us
W20250112 20:51:28.413892 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b1f4ea776424da59c8ec69715c5337d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.305990 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d4f060e4fa3648d6943cc97b9b3b7f51" } modify_external_catalogs: true
I20250112 20:51:28.415411 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d4f060e4fa3648d6943cc97b9b3b7f51" } modify_external_catalogs: true
I20250112 20:51:28.306350 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_11e0da2a1beb4196933adeeb88dfed4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.416231 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11e0da2a1beb4196933adeeb88dfed4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.033478 1164 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:28.417253 1164 ts_tablet_manager.cc:1397] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 1.147s user 0.005s sys 0.008s
I20250112 20:51:28.307600 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fb52d5971de44b71b7a53e1d05798b74" } modify_external_catalogs: true
I20250112 20:51:28.417618 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:27.959487 710 leader_election.cc:343] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.417711 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fb52d5971de44b71b7a53e1d05798b74" } modify_external_catalogs: true
W20250112 20:51:28.417838 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 465405us
I20250112 20:51:28.309903 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_07853a7e990447d293c9fd0530206abf" } new_table_name: "default.Table_b92d295d444f4b398aa900e96f1e0b99" modify_external_catalogs: true
I20250112 20:51:28.418085 1164 raft_consensus.cc:357] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.418303 1164 raft_consensus.cc:383] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.418407 1164 raft_consensus.cc:738] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.418692 1164 consensus_queue.cc:260] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.418809 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:28.312615 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71743c73b6f24da9b14a3d4148117813 consensus state has no leader
I20250112 20:51:28.419031 773 raft_consensus.cc:2463] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:28.419241 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 953d2ec8a7564c0bbcf0166487dd1746 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.419307 1164 ts_tablet_manager.cc:1428] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:28.419606 1164 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:28.419663 710 leader_election.cc:343] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.419740 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 0871d4272e874a958174f96bfc0e1a95 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.419770 710 leader_election.cc:304] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:28.305267 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_60ac48475bd54f50ba4f5028b07ec679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.421113 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60ac48475bd54f50ba4f5028b07ec679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.421466 1227 raft_consensus.cc:491] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:28.421620 1227 raft_consensus.cc:513] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.109731 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313a5c9905bc445893f1ffa8752ca589" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:28.422255 1227 leader_election.cc:290] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:28.422408 1227 raft_consensus.cc:2743] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:28.422712 1444 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:28.423635 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.433164 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cae067df1a40098f81cfbd17224c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.436491 1164 tablet_bootstrap.cc:654] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.441098 847 leader_election.cc:304] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:28.441398 1308 raft_consensus.cc:2798] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:28.441499 1308 raft_consensus.cc:491] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:28.441606 1308 raft_consensus.cc:3054] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:28.443130 983 consensus_peers.cc:487] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1dffbe43ba4eeab7ee12d901b62ffb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.444178 1308 raft_consensus.cc:513] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.444345 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 122) took 1858 ms. Trace:
I20250112 20:51:28.444432 1186 rpcz_store.cc:276] 0112 20:51:26.585464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.585502 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:27.657747 (+1072245us) ts_tablet_manager.cc:1145] Deleting tablet 8c7a94b2e9be4b5db3dfd6129b03cd25
0112 20:51:27.657754 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:27.935068 (+277314us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:27.987607 (+ 52539us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.444286 (+456679us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1418,"tablet-delete.queue_time_us":1072077}
I20250112 20:51:28.444813 1186 tablet_replica.cc:331] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:28.444909 879 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.445020 1186 raft_consensus.cc:2238] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.445173 1186 raft_consensus.cc:2267] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.445149 1436 consensus_queue.cc:1035] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.445441 1308 leader_election.cc:290] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:28.445619 1186 ts_tablet_manager.cc:1905] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.446074 1444 tablet.cc:1721] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:28.447181 1178 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:28.448792 1178 tablet_bootstrap.cc:654] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:28.450469 847 consensus_peers.cc:487] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.450502 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:28.451076 847 leader_election.cc:343] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.453603 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:28.453816 773 raft_consensus.cc:3054] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:28.456584 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e72fda1743fc4f96b29add6e36e884de consensus state has no leader
I20250112 20:51:28.456924 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e7a8468369af418ea4a1b0a681f8ff74 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.457077 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a692cc5fb6f4afc854c4821274edfdb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:28.458249 983 leader_election.cc:343] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 719b906973a8490898d6befce49910d3
I20250112 20:51:28.458316 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ea7336d13ea46748167e35e7e7e8c98 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.458400 983 leader_election.cc:304] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:28.458822 983 leader_election.cc:343] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 313a5c9905bc445893f1ffa8752ca589
I20250112 20:51:28.458878 453 master_service.cc:946] ReplaceTablet: received request to replace tablet e0f1c76db0d64624bd43f0e538f0649d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.458954 983 leader_election.cc:304] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:28.459209 1440 raft_consensus.cc:2743] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:28.459358 1354 raft_consensus.cc:2743] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:28.460006 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a692cc5fb6f4afc854c4821274edfdb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:28.461108 710 leader_election.cc:343] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:28.461371 710 leader_election.cc:343] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.461496 710 leader_election.cc:304] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:28.461688 1427 raft_consensus.cc:491] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:28.461808 1427 raft_consensus.cc:513] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.462453 1427 leader_election.cc:290] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:28.462611 1427 raft_consensus.cc:2743] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:28.463361 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b70ffb3e3341d398d36ff76993e9ac" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:28.463536 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b70ffb3e3341d398d36ff76993e9ac" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:28.463842 710 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:28.464113 710 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.464203 710 leader_election.cc:304] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:28.464385 847 consensus_peers.cc:487] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 313a5c9905bc445893f1ffa8752ca589. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.464416 1428 raft_consensus.cc:2743] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:28.464591 847 leader_election.cc:343] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:28.464700 847 leader_election.cc:304] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:28.471977 1184 ts_tablet_manager.cc:1918] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.472126 1184 log.cc:1198] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b07cef7306074ba3b7f8606bdfa6204a
I20250112 20:51:28.477666 1497 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:28.478097 1436 raft_consensus.cc:2743] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:28.479993 1505 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:28.480495 1185 ts_tablet_manager.cc:1918] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.480623 1185 log.cc:1198] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:28.497224 1185 ts_tablet_manager.cc:1939] T 5ea7336d13ea46748167e35e7e7e8c98 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:28.498088 773 raft_consensus.cc:2463] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:28.500734 847 leader_election.cc:304] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:28.501392 1430 raft_consensus.cc:2798] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:28.501549 1430 raft_consensus.cc:695] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:28.501869 1430 consensus_queue.cc:237] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.510428 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 135) took 1393 ms. Trace:
I20250112 20:51:28.510601 1185 rpcz_store.cc:276] 0112 20:51:27.117418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.117501 (+ 83us) service_pool.cc:225] Handling call
0112 20:51:27.835913 (+718412us) ts_tablet_manager.cc:1145] Deleting tablet 5ea7336d13ea46748167e35e7e7e8c98
0112 20:51:27.835920 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.028295 (+192375us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.480334 (+452039us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.510376 (+ 30042us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":718165}
I20250112 20:51:28.512130 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5ea7336d13ea46748167e35e7e7e8c98 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]) successfully deleted
I20250112 20:51:28.512506 1185 tablet_replica.cc:331] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:28.512763 1185 raft_consensus.cc:2238] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.513022 1185 raft_consensus.cc:2267] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.513677 1185 ts_tablet_manager.cc:1905] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.533643 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_673a2a9243984736b6b43018eb3e1a42 [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 2 got 1
I20250112 20:51:28.534111 454 catalog_manager.cc:5526] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:28.534483 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:28.535876 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0092258e094a38ba7e223be03d4c9a (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.536123 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0092258e094a38ba7e223be03d4c9a (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.536275 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0092258e094a38ba7e223be03d4c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.536474 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0092258e094a38ba7e223be03d4c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.537226 879 tablet_service.cc:1467] Processing CreateTablet for tablet 55d75f9b410f4790ac926646a3cf0fd0 (DEFAULT_TABLE table=default.Table_a476af27be9d446dab54b21d797ca38f [id=1ef2730aa400453a9d078c520a5329f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.538038 756 tablet_service.cc:1467] Processing CreateTablet for tablet 55d75f9b410f4790ac926646a3cf0fd0 (DEFAULT_TABLE table=default.Table_a476af27be9d446dab54b21d797ca38f [id=1ef2730aa400453a9d078c520a5329f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.538362 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d75f9b410f4790ac926646a3cf0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:28.541105 710 consensus_peers.cc:487] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.541049 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0092258e094a38ba7e223be03d4c9a (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.548965 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 55d75f9b410f4790ac926646a3cf0fd0 (DEFAULT_TABLE table=default.Table_a476af27be9d446dab54b21d797ca38f [id=1ef2730aa400453a9d078c520a5329f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.549470 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d75f9b410f4790ac926646a3cf0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.554494 1184 ts_tablet_manager.cc:1939] T b07cef7306074ba3b7f8606bdfa6204a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:28.555482 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 122) took 1918 ms. Trace:
I20250112 20:51:28.555595 1184 rpcz_store.cc:276] 0112 20:51:26.637254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.638792 (+ 1538us) service_pool.cc:225] Handling call
0112 20:51:27.894710 (+1255918us) ts_tablet_manager.cc:1145] Deleting tablet b07cef7306074ba3b7f8606bdfa6204a
0112 20:51:27.894721 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.041335 (+146614us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.471854 (+430519us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.555415 (+ 83561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255688}
I20250112 20:51:28.556126 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b07cef7306074ba3b7f8606bdfa6204a (table default.Table_b92d295d444f4b398aa900e96f1e0b99 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:28.556496 1184 tablet_replica.cc:331] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:28.556733 1184 raft_consensus.cc:2238] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.556962 1184 raft_consensus.cc:2267] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.557610 1184 ts_tablet_manager.cc:1905] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.583577 1451 sys_catalog.cc:458] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:28.584841 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0871d4272e874a958174f96bfc0e1a95
I20250112 20:51:28.585290 446 catalog_manager.cc:6596] ReplaceTablet: tablet 0871d4272e874a958174f96bfc0e1a95 deleted and replaced by tablet 3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:28.586385 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.586918 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.587064 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0871d4272e874a958174f96bfc0e1a95 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b7f79b3309e45afb9e18cad97ddfeb0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:28.587064 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0871d4272e874a958174f96bfc0e1a95 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b7f79b3309e45afb9e18cad97ddfeb0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.587368 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0871d4272e874a958174f96bfc0e1a95 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b7f79b3309e45afb9e18cad97ddfeb0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.592134 1451 raft_consensus.cc:491] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:28.592450 1451 raft_consensus.cc:513] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:28.604055 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:28.606184 1441 consensus_queue.cc:1035] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.608299 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a8f2b2f1a79c4ad6a6defba7d45b05c7" } new_table_name: "default.Table_9c771feed27441ec8340f7279a4a06c1" modify_external_catalogs: true
I20250112 20:51:28.609400 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6f4421d16e3f423794c44970a2469b10
I20250112 20:51:28.610348 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 201a7e2f17bb46fa918dad8f878a4065
I20250112 20:51:28.612001 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:28.613147 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.613386 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.613443 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8d5818f34258400982f08f02014c11bf" } new_table_name: "default.Table_0afc27e58eec42daadf528a66340c624" modify_external_catalogs: true
I20250112 20:51:28.614980 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_47f89b433c7e4b11a6e685b3c702c3d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:28.615274 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:28.615510 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f89b433c7e4b11a6e685b3c702c3d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.615619 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.615880 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.617319 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:28.619302 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f249d70b116443b7823e093efeda5cc3
I20250112 20:51:28.619376 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1c76db0d64624bd43f0e538f0649d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.619686 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1c76db0d64624bd43f0e538f0649d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:28.621086 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0fd03aba45a64415a098af438c178bc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.621603 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fd03aba45a64415a098af438c178bc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.620373 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1c76db0d64624bd43f0e538f0649d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.622154 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.622426 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:28.622691 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:28.622884 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_66f53100719d424aa91c8c1ddad82c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.623379 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66f53100719d424aa91c8c1ddad82c8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.624670 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_dc0bcf302ab44599b97fbba9f0301582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.625192 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc0bcf302ab44599b97fbba9f0301582 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.628994 1451 leader_election.cc:290] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153)
I20250112 20:51:28.631217 1440 consensus_queue.cc:1035] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.657721 1163 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:28.657899 1163 ts_tablet_manager.cc:1397] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.252s user 0.025s sys 0.038s
I20250112 20:51:28.658697 1163 raft_consensus.cc:357] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.658913 1163 raft_consensus.cc:383] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.658989 1163 raft_consensus.cc:738] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.659255 1163 consensus_queue.cc:260] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.659770 1163 ts_tablet_manager.cc:1428] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:28.660063 1163 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:28.662508 1163 tablet_bootstrap.cc:654] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.680714 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d75f9b410f4790ac926646a3cf0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.681465 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0092258e094a38ba7e223be03d4c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.692420 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.692698 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.700887 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_40c44749c3934a83ab1b8ed501bf1c91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.701593 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40c44749c3934a83ab1b8ed501bf1c91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.704224 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_60ac48475bd54f50ba4f5028b07ec679" } modify_external_catalogs: true
I20250112 20:51:28.704447 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_60ac48475bd54f50ba4f5028b07ec679" } modify_external_catalogs: true
W20250112 20:51:28.707144 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4390c60d6c724190a216ef9c961b9592 consensus state has no leader
W20250112 20:51:28.707839 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64cd83b199bb4da28d39da449945c389 consensus state has no leader
I20250112 20:51:28.708597 1495 consensus_queue.cc:1035] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.710278 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1bea96e02dff4af98e79dc9e9ab83956" } new_table_name: "default.Table_4c02bae8e4d64469b992b13dd36c7e81" modify_external_catalogs: true
I20250112 20:51:28.712226 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:28.712467 1186 ts_tablet_manager.cc:1918] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.712610 1186 log.cc:1198] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/30e53e67076c4a5a89f1e93fc168a997
W20250112 20:51:28.712658 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.713083 1186 ts_tablet_manager.cc:1939] T 30e53e67076c4a5a89f1e93fc168a997 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:28.713492 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 120) took 2128 ms. Trace:
I20250112 20:51:28.717090 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6b82fb5b4ad04365963324b64a52c22b" } new_table_name: "default.Table_43f4ea15650c4619afdabdb3c169df86" modify_external_catalogs: true
I20250112 20:51:28.801532 1531 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 20:51:28.794782 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b985fd656ce44131bc91ccfeb79587c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "0f5c09e899184de6b770dca873501062" is_pre_election: true
I20250112 20:51:28.720566 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2a522da7a096468db3231e0f9f8205bc" } new_table_name: "default.Table_196bfe6e0c8e4f50a711232f69d23c87" modify_external_catalogs: true
I20250112 20:51:28.729352 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_74308835319d415ba7a13ff9e0a42d76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:28.729738 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_395d5be237c54baea392437bef2cbadf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.841325 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74308835319d415ba7a13ff9e0a42d76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.713632 1186 rpcz_store.cc:276] 0112 20:51:26.584952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.586736 (+ 1784us) service_pool.cc:225] Handling call
0112 20:51:28.444699 (+1857963us) ts_tablet_manager.cc:1145] Deleting tablet 30e53e67076c4a5a89f1e93fc168a997
0112 20:51:28.444713 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.524009 (+ 79296us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.712330 (+188321us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.713458 (+ 1128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1857505}
W20250112 20:51:28.841352 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_395d5be237c54baea392437bef2cbadf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.841945 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 118) took 2261 ms. Trace:
I20250112 20:51:28.842069 1186 rpcz_store.cc:276] 0112 20:51:26.580289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.596147 (+ 15858us) service_pool.cc:225] Handling call
0112 20:51:28.841762 (+2245615us) ts_tablet_manager.cc:1145] Deleting tablet 158cfe5ba52c4771af97841885b29f9d
0112 20:51:28.841777 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.841900 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245351}
I20250112 20:51:28.842446 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 123) took 2254 ms. Trace:
I20250112 20:51:28.842550 1186 rpcz_store.cc:276] 0112 20:51:26.588182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.600752 (+ 12570us) service_pool.cc:225] Handling call
0112 20:51:28.842317 (+2241565us) ts_tablet_manager.cc:1145] Deleting tablet 7f39195945024f4fba9942faec70bc31
0112 20:51:28.842326 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.842414 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239425}
I20250112 20:51:28.842897 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 117) took 2263 ms. Trace:
I20250112 20:51:28.842989 1186 rpcz_store.cc:276] 0112 20:51:26.579495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.603459 (+ 23964us) service_pool.cc:225] Handling call
0112 20:51:28.842780 (+2239321us) ts_tablet_manager.cc:1145] Deleting tablet 29b70ffb3e3341d398d36ff76993e9ac
0112 20:51:28.842789 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.842870 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239036}
I20250112 20:51:28.399574 1357 rpcz_store.cc:276] 0112 20:51:27.166185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.166350 (+ 165us) service_pool.cc:225] Handling call
0112 20:51:28.018268 (+851918us) raft_consensus.cc:1466] Updating replica for 179 ops
0112 20:51:28.090615 (+ 72347us) raft_consensus.cc:1529] Early marking committed up to index 27
0112 20:51:28.128177 (+ 37562us) raft_consensus.cc:1534] Triggering prepare for 179 ops
0112 20:51:28.128531 (+ 354us) write_op.cc:270] Start()
0112 20:51:28.128578 (+ 47us) write_op.cc:276] Timestamp: P: 1736715083896351 usec, L: 0
0112 20:51:28.129336 (+ 758us) write_op.cc:270] Start()
0112 20:51:28.129380 (+ 44us) write_op.cc:276] Timestamp: P: 1736715083897137 usec, L: 0
0112 20:51:28.129587 (+ 207us) write_op.cc:270] Start()
0112 20:51:28.129607 (+ 20us) write_op.cc:276] Timestamp: P: 1736715083920003 usec, L: 0
0112 20:51:28.129752 (+ 145us) write_op.cc:270] Start()
0112 20:51:28.129772 (+ 20us) write_op.cc:276] Timestamp: P: 1736715083920835 usec, L: 0
0112 20:51:28.134895 (+ 5123us) write_op.cc:270] Start()
0112 20:51:28.134922 (+ 27us) write_op.cc:276] Timestamp: P: 1736715083921489 usec, L: 0
0112 20:51:28.135090 (+ 168us) write_op.cc:270] Start()
0112 20:51:28.135115 (+ 25us) write_op.cc:276] Timestamp: P: 1736715083922118 usec, L: 0
0112 20:51:28.135259 (+ 144us) write_op.cc:270] Start()
0112 20:51:28.135298 (+ 39us) write_op.cc:276] Timestamp: P: 1736715083922743 usec, L: 0
0112 20:51:28.135415 (+ 117us) write_op.cc:270] Start()
0112 20:51:28.135432 (+ 17us) write_op.cc:276] Timestamp: P: 1736715083923324 usec, L: 0
0112 20:51:28.135579 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.135612 (+ 33us) write_op.cc:276] Timestamp: P: 1736715084039286 usec, L: 0
0112 20:51:28.135818 (+ 206us) write_op.cc:270] Start()
0112 20:51:28.135837 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084040728 usec, L: 0
0112 20:51:28.135959 (+ 122us) write_op.cc:270] Start()
0112 20:51:28.135977 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084041836 usec, L: 0
0112 20:51:28.136130 (+ 153us) write_op.cc:270] Start()
0112 20:51:28.136147 (+ 17us) write_op.cc:276] Timestamp: P: 1736715084042820 usec, L: 0
0112 20:51:28.136274 (+ 127us) write_op.cc:270] Start()
0112 20:51:28.136289 (+ 15us) write_op.cc:276] Timestamp: P: 1736715084067508 usec, L: 0
0112 20:51:28.136419 (+ 130us) write_op.cc:270] Start()
0112 20:51:28.136447 (+ 28us) write_op.cc:276] Timestamp: P: 1736715084106832 usec, L: 0
0112 20:51:28.136591 (+ 144us) write_op.cc:270] Start()
0112 20:51:28.136617 (+ 26us) write_op.cc:276] Timestamp: P: 1736715084164858 usec, L: 0
0112 20:51:28.136733 (+ 116us) write_op.cc:270] Start()
0112 20:51:28.136764 (+ 31us) write_op.cc:276] Timestamp: P: 1736715084175827 usec, L: 0
0112 20:51:28.136943 (+ 179us) write_op.cc:270] Start()
0112 20:51:28.136961 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084177360 usec, L: 0
0112 20:51:28.137091 (+ 130us) write_op.cc:270] Start()
0112 20:51:28.137107 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084178209 usec, L: 0
0112 20:51:28.137228 (+ 121us) write_op.cc:270] Start()
0112 20:51:28.137244 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084203680 usec, L: 0
0112 20:51:28.137366 (+ 122us) write_op.cc:270] Start()
0112 20:51:28.137381 (+ 15us) write_op.cc:276] Timestamp: P: 1736715084206073 usec, L: 0
0112 20:51:28.137555 (+ 174us) write_op.cc:270] Start()
0112 20:51:28.137574 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084212445 usec, L: 0
0112 20:51:28.137716 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.137734 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084215518 usec, L: 0
0112 20:51:28.137924 (+ 190us) write_op.cc:270] Start()
0112 20:51:28.137947 (+ 23us) write_op.cc:276] Timestamp: P: 1736715084220613 usec, L: 0
0112 20:51:28.138064 (+ 117us) write_op.cc:270] Start()
0112 20:51:28.138086 (+ 22us) write_op.cc:276] Timestamp: P: 1736715084222448 usec, L: 0
0112 20:51:28.138244 (+ 158us) write_op.cc:270] Start()
0112 20:51:28.138261 (+ 17us) write_op.cc:276] Timestamp: P: 1736715084229741 usec, L: 0
0112 20:51:28.138382 (+ 121us) write_op.cc:270] Start()
0112 20:51:28.138398 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084231319 usec, L: 0
0112 20:51:28.138510 (+ 112us) write_op.cc:270] Start()
0112 20:51:28.138535 (+ 25us) write_op.cc:276] Timestamp: P: 1736715084232907 usec, L: 0
0112 20:51:28.138682 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.138704 (+ 22us) write_op.cc:276] Timestamp: P: 1736715084298693 usec, L: 0
0112 20:51:28.138821 (+ 117us) write_op.cc:270] Start()
0112 20:51:28.138842 (+ 21us) write_op.cc:276] Timestamp: P: 1736715084300784 usec, L: 0
0112 20:51:28.138985 (+ 143us) write_op.cc:270] Start()
0112 20:51:28.139005 (+ 20us) write_op.cc:276] Timestamp: P: 1736715084301787 usec, L: 0
0112 20:51:28.139119 (+ 114us) write_op.cc:270] Start()
0112 20:51:28.139140 (+ 21us) write_op.cc:276] Timestamp: P: 1736715084302796 usec, L: 0
0112 20:51:28.139242 (+ 102us) write_op.cc:270] Start()
0112 20:51:28.139261 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084329102 usec, L: 0
0112 20:51:28.139383 (+ 122us) write_op.cc:270] Start()
0112 20:51:28.139402 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084329705 usec, L: 0
0112 20:51:28.139541 (+ 139us) write_op.cc:270] Start()
0112 20:51:28.139576 (+ 35us) write_op.cc:276] Timestamp: P: 1736715084341354 usec, L: 0
0112 20:51:28.139705 (+ 129us) write_op.cc:270] Start()
0112 20:51:28.139723 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084341906 usec, L: 0
0112 20:51:28.139838 (+ 115us) write_op.cc:270] Start()
0112 20:51:28.139888 (+ 50us) write_op.cc:276] Timestamp: P: 1736715084356941 usec, L: 0
0112 20:51:28.140037 (+ 149us) write_op.cc:270] Start()
0112 20:51:28.140056 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084379510 usec, L: 0
0112 20:51:28.140174 (+ 118us) write_op.cc:270] Start()
0112 20:51:28.140191 (+ 17us) write_op.cc:276] Timestamp: P: 1736715084392487 usec, L: 0
0112 20:51:28.140311 (+ 120us) write_op.cc:270] Start()
0112 20:51:28.140326 (+ 15us) write_op.cc:276] Timestamp: P: 1736715084414124 usec, L: 0
0112 20:51:28.140477 (+ 151us) write_op.cc:270] Start()
0112 20:51:28.140498 (+ 21us) write_op.cc:276] Timestamp: P: 1736715084478773 usec, L: 0
0112 20:51:28.140626 (+ 128us) write_op.cc:270] Start()
0112 20:51:28.140646 (+ 20us) write_op.cc:276] Timestamp: P: 1736715084494768 usec, L: 0
0112 20:51:28.140780 (+ 134us) write_op.cc:270] Start()
0112 20:51:28.141211 (+ 431us) write_op.cc:276] Timestamp: P: 1736715084500506 usec, L: 0
0112 20:51:28.141340 (+ 129us) write_op.cc:270] Start()
0112 20:51:28.141361 (+ 21us) write_op.cc:276] Timestamp: P: 1736715084503194 usec, L: 0
0112 20:51:28.141499 (+ 138us) write_op.cc:270] Start()
0112 20:51:28.141515 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084521156 usec, L: 0
0112 20:51:28.141630 (+ 115us) write_op.cc:270] Start()
0112 20:51:28.141655 (+ 25us) write_op.cc:276] Timestamp: P: 1736715084535084 usec, L: 0
0112 20:51:28.141796 (+ 141us) write_op.cc:270] Start()
0112 20:51:28.141812 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084542130 usec, L: 0
0112 20:51:28.141944 (+ 132us) write_op.cc:270] Start()
0112 20:51:28.141959 (+ 15us) write_op.cc:276] Timestamp: P: 1736715084554707 usec, L: 0
0112 20:51:28.142098 (+ 139us) write_op.cc:270] Start()
0112 20:51:28.142118 (+ 20us) write_op.cc:276] Timestamp: P: 1736715084560202 usec, L: 0
0112 20:51:28.142235 (+ 117us) write_op.cc:270] Start()
0112 20:51:28.142254 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084572634 usec, L: 0
0112 20:51:28.142425 (+ 171us) write_op.cc:270] Start()
0112 20:51:28.142442 (+ 17us) write_op.cc:276] Timestamp: P: 1736715084574820 usec, L: 0
0112 20:51:28.142562 (+ 120us) write_op.cc:270] Start()
0112 20:51:28.142577 (+ 15us) write_op.cc:276] Timestamp: P: 1736715084581971 usec, L: 0
0112 20:51:28.142702 (+ 125us) write_op.cc:270] Start()
0112 20:51:28.142718 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084596707 usec, L: 0
0112 20:51:28.142845 (+ 127us) write_op.cc:270] Start()
0112 20:51:28.142861 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084597333 usec, L: 0
0112 20:51:28.142981 (+ 120us) write_op.cc:270] Start()
0112 20:51:28.142997 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084597958 usec, L: 0
0112 20:51:28.143107 (+ 110us) write_op.cc:270] Start()
0112 20:51:28.143125 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084598488 usec, L: 0
0112 20:51:28.143246 (+ 121us) write_op.cc:270] Start()
0112 20:51:28.143264 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084599236 usec, L: 0
0112 20:51:28.143370 (+ 106us) write_op.cc:270] Start()
0112 20:51:28.143388 (+ 18us) write_op.cc:276] Timestamp: P: 1736715084600598 usec, L: 0
0112 20:51:28.143512 (+ 124us) write_op.cc:270] Start()
0112 20:51:28.143533 (+ 21us) write_op.cc:276] Timestamp: P: 1736715084601574 usec, L: 0
0112 20:51:28.143677 (+ 144us) write_op.cc:270] Start()
0112 20:51:28.143700 (+ 23us) write_op.cc:276] Timestamp: P: 1736715084721185 usec, L: 0
0112 20:51:28.143829 (+ 129us) write_op.cc:270] Start()
0112 20:51:28.143868 (+ 39us) write_op.cc:276] Timestamp: P: 1736715084732587 usec, L: 0
0112 20:51:28.143996 (+ 128us) write_op.cc:270] Start()
0112 20:51:28.144012 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084738931 usec, L: 0
0112 20:51:28.144135 (+ 123us) write_op.cc:270] Start()
0112 20:51:28.144151 (+ 16us) write_op.cc:276] Timestamp: P: 1736715084741414 usec, L: 0
0112 20:51:28.144277 (+ 126us) write_op.cc:270] Start()
0112 20:51:28.144296 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084741910 usec, L: 0
0112 20:51:28.144464 (+ 168us) write_op.cc:270] Start()
0112 20:51:28.144488 (+ 24us) write_op.cc:276] Timestamp: P: 1736715084742383 usec, L: 0
0112 20:51:28.144630 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.144655 (+ 25us) write_op.cc:276] Timestamp: P: 1736715084742879 usec, L: 0
0112 20:51:28.150637 (+ 5982us) write_op.cc:270] Start()
0112 20:51:28.150679 (+ 42us) write_op.cc:276] Timestamp: P: 1736715084743490 usec, L: 0
0112 20:51:28.150830 (+ 151us) write_op.cc:270] Start()
0112 20:51:28.150849 (+ 19us) write_op.cc:276] Timestamp: P: 1736715084807014 usec, L: 0
0112 20:51:28.151002 (+ 153us) write_op.cc:270] Start()
0112 20:51:28.151024 (+ 22us) write_op.cc:276] Timestamp: P: 1736715084824242 usec, L: 0
0112 20:51:28.151158 (+ 134us) write_op.cc:270] Start()
0112 20:51:28.151182 (+ 24us) write_op.cc:276] Timestamp: P: 1736715084850046 usec, L: 0
0112 20:51:28.151380 (+ 198us) write_op.cc:270] Start()
0112 20:51:28.151406 (+ 26us) write_op.cc:276] Timestamp: P: 1736715084850661 usec, L: 0
0112 20:51:28.151568 (+ 162us) write_op.cc:270] Start()
0112 20:51:28.151591 (+ 23us) write_op.cc:276] Timestamp: P: 1736715084851246 usec, L: 0
0112 20:51:28.151767 (+ 176us) write_op.cc:270] Start()
0112 20:51:28.151791 (+ 24us) write_op.cc:276] Timestamp: P: 1736715084851746 usec, L: 0
0112 20:51:28.151922 (+ 131us) write_op.cc:270] Start()
0112 20:51:28.151944 (+ 22us) write_op.cc:276] Timestamp: P: 1736715084854780 usec, L: 0
0112 20:51:28.152105 (+ 161us) write_op.cc:270] Start()
0112 20:51:28.152127 (+ 22us) write_op.cc:276] Timestamp: P: 1736715084966033 usec, L: 0
0112 20:51:28.152286 (+ 159us) write_op.cc:270] Start()
0112 20:51:28.152308 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085093682 usec, L: 0
0112 20:51:28.152489 (+ 181us) write_op.cc:270] Start()
0112 20:51:28.152508 (+ 19us) write_op.cc:276] Timestamp: P: 1736715085105543 usec, L: 0
0112 20:51:28.152661 (+ 153us) write_op.cc:270] Start()
0112 20:51:28.152682 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085107786 usec, L: 0
0112 20:51:28.152862 (+ 180us) write_op.cc:270] Start()
0112 20:51:28.152885 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085108514 usec, L: 0
0112 20:51:28.152994 (+ 109us) write_op.cc:270] Start()
0112 20:51:28.153035 (+ 41us) write_op.cc:276] Timestamp: P: 1736715085109373 usec, L: 0
0112 20:51:28.153202 (+ 167us) write_op.cc:270] Start()
0112 20:51:28.153223 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085110224 usec, L: 0
0112 20:51:28.153391 (+ 168us) write_op.cc:270] Start()
0112 20:51:28.153413 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085141406 usec, L: 0
0112 20:51:28.153574 (+ 161us) write_op.cc:270] Start()
0112 20:51:28.153607 (+ 33us) write_op.cc:276] Timestamp: P: 1736715085143677 usec, L: 0
0112 20:51:28.153778 (+ 171us) write_op.cc:270] Start()
0112 20:51:28.153801 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085159758 usec, L: 0
0112 20:51:28.153941 (+ 140us) write_op.cc:270] Start()
0112 20:51:28.153964 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085184123 usec, L: 0
0112 20:51:28.154106 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.154127 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085186166 usec, L: 0
0112 20:51:28.154269 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.154291 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085245356 usec, L: 0
0112 20:51:28.154438 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.154461 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085247592 usec, L: 0
0112 20:51:28.154618 (+ 157us) write_op.cc:270] Start()
0112 20:51:28.154661 (+ 43us) write_op.cc:276] Timestamp: P: 1736715085248783 usec, L: 0
0112 20:51:28.154820 (+ 159us) write_op.cc:270] Start()
0112 20:51:28.154838 (+ 18us) write_op.cc:276] Timestamp: P: 1736715085300451 usec, L: 0
0112 20:51:28.154966 (+ 128us) write_op.cc:270] Start()
0112 20:51:28.154985 (+ 19us) write_op.cc:276] Timestamp: P: 1736715085303858 usec, L: 0
0112 20:51:28.155131 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.155152 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085426096 usec, L: 0
0112 20:51:28.155276 (+ 124us) write_op.cc:270] Start()
0112 20:51:28.155298 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085435282 usec, L: 0
0112 20:51:28.155434 (+ 136us) write_op.cc:270] Start()
0112 20:51:28.155455 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085444687 usec, L: 0
0112 20:51:28.155590 (+ 135us) write_op.cc:270] Start()
0112 20:51:28.155609 (+ 19us) write_op.cc:276] Timestamp: P: 1736715085576704 usec, L: 0
0112 20:51:28.155732 (+ 123us) write_op.cc:270] Start()
0112 20:51:28.155751 (+ 19us) write_op.cc:276] Timestamp: P: 1736715085577513 usec, L: 0
0112 20:51:28.155923 (+ 172us) write_op.cc:270] Start()
0112 20:51:28.155945 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085577961 usec, L: 0
0112 20:51:28.156076 (+ 131us) write_op.cc:270] Start()
0112 20:51:28.156095 (+ 19us) write_op.cc:276] Timestamp: P: 1736715085671535 usec, L: 0
0112 20:51:28.156219 (+ 124us) write_op.cc:270] Start()
0112 20:51:28.156239 (+ 20us) write_op.cc:276] Timestamp: P: 1736715085673756 usec, L: 0
0112 20:51:28.156402 (+ 163us) write_op.cc:270] Start()
0112 20:51:28.156422 (+ 20us) write_op.cc:276] Timestamp: P: 1736715085675163 usec, L: 0
0112 20:51:28.156551 (+ 129us) write_op.cc:270] Start()
0112 20:51:28.156579 (+ 28us) write_op.cc:276] Timestamp: P: 1736715085676080 usec, L: 0
0112 20:51:28.156713 (+ 134us) write_op.cc:270] Start()
0112 20:51:28.156733 (+ 20us) write_op.cc:276] Timestamp: P: 1736715085720890 usec, L: 0
0112 20:51:28.156932 (+ 199us) write_op.cc:270] Start()
0112 20:51:28.156954 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085723332 usec, L: 0
0112 20:51:28.157087 (+ 133us) write_op.cc:270] Start()
0112 20:51:28.157109 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085833858 usec, L: 0
0112 20:51:28.157248 (+ 139us) write_op.cc:270] Start()
0112 20:51:28.157271 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085865496 usec, L: 0
0112 20:51:28.157414 (+ 143us) write_op.cc:270] Start()
0112 20:51:28.157436 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085875267 usec, L: 0
0112 20:51:28.157575 (+ 139us) write_op.cc:270] Start()
0112 20:51:28.157596 (+ 21us) write_op.cc:276] Timestamp: P: 1736715085883657 usec, L: 0
0112 20:51:28.157758 (+ 162us) write_op.cc:270] Start()
0112 20:51:28.157780 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085905017 usec, L: 0
0112 20:51:28.157911 (+ 131us) write_op.cc:270] Start()
0112 20:51:28.157933 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085935567 usec, L: 0
0112 20:51:28.158075 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.158114 (+ 39us) write_op.cc:276] Timestamp: P: 1736715085947750 usec, L: 0
0112 20:51:28.158278 (+ 164us) write_op.cc:270] Start()
0112 20:51:28.158301 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085957299 usec, L: 0
0112 20:51:28.158439 (+ 138us) write_op.cc:270] Start()
0112 20:51:28.158462 (+ 23us) write_op.cc:276] Timestamp: P: 1736715085970805 usec, L: 0
0112 20:51:28.158768 (+ 306us) write_op.cc:270] Start()
0112 20:51:28.158800 (+ 32us) write_op.cc:276] Timestamp: P: 1736715085971443 usec, L: 0
0112 20:51:28.158946 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.158968 (+ 22us) write_op.cc:276] Timestamp: P: 1736715085984419 usec, L: 0
0112 20:51:28.159114 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.159136 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086013918 usec, L: 0
0112 20:51:28.159322 (+ 186us) write_op.cc:270] Start()
0112 20:51:28.159346 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086032538 usec, L: 0
0112 20:51:28.159490 (+ 144us) write_op.cc:270] Start()
0112 20:51:28.159527 (+ 37us) write_op.cc:276] Timestamp: P: 1736715086056865 usec, L: 0
0112 20:51:28.159704 (+ 177us) write_op.cc:270] Start()
0112 20:51:28.159727 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086076854 usec, L: 0
0112 20:51:28.159873 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.159897 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086079111 usec, L: 0
0112 20:51:28.160045 (+ 148us) write_op.cc:270] Start()
0112 20:51:28.160066 (+ 21us) write_op.cc:276] Timestamp: P: 1736715086087046 usec, L: 0
0112 20:51:28.160213 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.160236 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086087594 usec, L: 0
0112 20:51:28.160360 (+ 124us) write_op.cc:270] Start()
0112 20:51:28.160380 (+ 20us) write_op.cc:276] Timestamp: P: 1736715086183586 usec, L: 0
0112 20:51:28.160521 (+ 141us) write_op.cc:270] Start()
0112 20:51:28.160545 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086184422 usec, L: 0
0112 20:51:28.160724 (+ 179us) write_op.cc:270] Start()
0112 20:51:28.160748 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086188308 usec, L: 0
0112 20:51:28.160909 (+ 161us) write_op.cc:270] Start()
0112 20:51:28.160927 (+ 18us) write_op.cc:276] Timestamp: P: 1736715086206017 usec, L: 0
0112 20:51:28.161152 (+ 225us) write_op.cc:270] Start()
0112 20:51:28.161177 (+ 25us) write_op.cc:276] Timestamp: P: 1736715086327568 usec, L: 0
0112 20:51:28.161319 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.161338 (+ 19us) write_op.cc:276] Timestamp: P: 1736715086330496 usec, L: 0
0112 20:51:28.161509 (+ 171us) write_op.cc:270] Start()
0112 20:51:28.161531 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086390738 usec, L: 0
0112 20:51:28.161726 (+ 195us) write_op.cc:270] Start()
0112 20:51:28.161749 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086391966 usec, L: 0
0112 20:51:28.161951 (+ 202us) write_op.cc:270] Start()
0112 20:51:28.161975 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086393286 usec, L: 0
0112 20:51:28.162136 (+ 161us) write_op.cc:270] Start()
0112 20:51:28.162158 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086404726 usec, L: 0
0112 20:51:28.162369 (+ 211us) write_op.cc:270] Start()
0112 20:51:28.162418 (+ 49us) write_op.cc:276] Timestamp: P: 1736715086408481 usec, L: 0
0112 20:51:28.162565 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.162587 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086411058 usec, L: 0
0112 20:51:28.162734 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.162757 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086417452 usec, L: 0
0112 20:51:28.162903 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.162925 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086448863 usec, L: 0
0112 20:51:28.163065 (+ 140us) write_op.cc:270] Start()
0112 20:51:28.163088 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086449588 usec, L: 0
0112 20:51:28.163230 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.163253 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086455317 usec, L: 0
0112 20:51:28.163402 (+ 149us) write_op.cc:270] Start()
0112 20:51:28.163423 (+ 21us) write_op.cc:276] Timestamp: P: 1736715086470205 usec, L: 0
0112 20:51:28.165060 (+ 1637us) write_op.cc:270] Start()
0112 20:51:28.165087 (+ 27us) write_op.cc:276] Timestamp: P: 1736715086490748 usec, L: 0
0112 20:51:28.165277 (+ 190us) write_op.cc:270] Start()
0112 20:51:28.165301 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086495011 usec, L: 0
0112 20:51:28.165444 (+ 143us) write_op.cc:270] Start()
0112 20:51:28.165467 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086510429 usec, L: 0
0112 20:51:28.165613 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.165635 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086550752 usec, L: 0
0112 20:51:28.165808 (+ 173us) write_op.cc:270] Start()
0112 20:51:28.165830 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086551981 usec, L: 0
0112 20:51:28.165970 (+ 140us) write_op.cc:270] Start()
0112 20:51:28.165985 (+ 15us) write_op.cc:276] Timestamp: P: 1736715086620484 usec, L: 0
0112 20:51:28.166127 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.166145 (+ 18us) write_op.cc:276] Timestamp: P: 1736715086665544 usec, L: 0
0112 20:51:28.166289 (+ 144us) write_op.cc:270] Start()
0112 20:51:28.166311 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086683132 usec, L: 0
0112 20:51:28.166448 (+ 137us) write_op.cc:270] Start()
0112 20:51:28.166473 (+ 25us) write_op.cc:276] Timestamp: P: 1736715086703068 usec, L: 0
0112 20:51:28.166606 (+ 133us) write_op.cc:270] Start()
0112 20:51:28.166628 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086721073 usec, L: 0
0112 20:51:28.166770 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.166792 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086748618 usec, L: 0
0112 20:51:28.166953 (+ 161us) write_op.cc:270] Start()
0112 20:51:28.166976 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086760827 usec, L: 0
0112 20:51:28.167138 (+ 162us) write_op.cc:270] Start()
0112 20:51:28.167161 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086771609 usec, L: 0
0112 20:51:28.167301 (+ 140us) write_op.cc:270] Start()
0112 20:51:28.167322 (+ 21us) write_op.cc:276] Timestamp: P: 1736715086783017 usec, L: 0
0112 20:51:28.167443 (+ 121us) write_op.cc:270] Start()
0112 20:51:28.167458 (+ 15us) write_op.cc:276] Timestamp: P: 1736715086793115 usec, L: 0
0112 20:51:28.167577 (+ 119us) write_op.cc:270] Start()
0112 20:51:28.167600 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086800651 usec, L: 0
0112 20:51:28.167803 (+ 203us) write_op.cc:270] Start()
0112 20:51:28.167827 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086816311 usec, L: 0
0112 20:51:28.167984 (+ 157us) write_op.cc:270] Start()
0112 20:51:28.168007 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086834585 usec, L: 0
0112 20:51:28.168139 (+ 132us) write_op.cc:270] Start()
0112 20:51:28.168161 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086911136 usec, L: 0
0112 20:51:28.168316 (+ 155us) write_op.cc:270] Start()
0112 20:51:28.168339 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086931325 usec, L: 0
0112 20:51:28.168487 (+ 148us) write_op.cc:270] Start()
0112 20:51:28.168511 (+ 24us) write_op.cc:276] Timestamp: P: 1736715086945319 usec, L: 0
0112 20:51:28.168646 (+ 135us) write_op.cc:270] Start()
0112 20:51:28.168680 (+ 34us) write_op.cc:276] Timestamp: P: 1736715086968437 usec, L: 0
0112 20:51:28.168843 (+ 163us) write_op.cc:270] Start()
0112 20:51:28.168870 (+ 27us) write_op.cc:276] Timestamp: P: 1736715086969423 usec, L: 0
0112 20:51:28.169016 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.169047 (+ 31us) write_op.cc:276] Timestamp: P: 1736715086970228 usec, L: 0
0112 20:51:28.169190 (+ 143us) write_op.cc:270] Start()
0112 20:51:28.169215 (+ 25us) write_op.cc:276] Timestamp: P: 1736715086970936 usec, L: 0
0112 20:51:28.169361 (+ 146us) write_op.cc:270] Start()
0112 20:51:28.169382 (+ 21us) write_op.cc:276] Timestamp: P: 1736715086971601 usec, L: 0
0112 20:51:28.169513 (+ 131us) write_op.cc:270] Start()
0112 20:51:28.169536 (+ 23us) write_op.cc:276] Timestamp: P: 1736715086972308 usec, L: 0
0112 20:51:28.169725 (+ 189us) write_op.cc:270] Start()
0112 20:51:28.169746 (+ 21us) write_op.cc:276] Timestamp: P: 1736715086983088 usec, L: 0
0112 20:51:28.169896 (+ 150us) write_op.cc:270] Start()
0112 20:51:28.169918 (+ 22us) write_op.cc:276] Timestamp: P: 1736715086983959 usec, L: 0
0112 20:51:28.170070 (+ 152us) write_op.cc:270] Start()
0112 20:51:28.170163 (+ 93us) write_op.cc:276] Timestamp: P: 1736715086992225 usec, L: 0
0112 20:51:28.170371 (+ 208us) write_op.cc:270] Start()
0112 20:51:28.170395 (+ 24us) write_op.cc:276] Timestamp: P: 1736715087004469 usec, L: 0
0112 20:51:28.170535 (+ 140us) write_op.cc:270] Start()
0112 20:51:28.170556 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087005490 usec, L: 0
0112 20:51:28.170753 (+ 197us) write_op.cc:270] Start()
0112 20:51:28.170775 (+ 22us) write_op.cc:276] Timestamp: P: 1736715087037073 usec, L: 0
0112 20:51:28.170923 (+ 148us) write_op.cc:270] Start()
0112 20:51:28.170947 (+ 24us) write_op.cc:276] Timestamp: P: 1736715087037991 usec, L: 0
0112 20:51:28.171089 (+ 142us) write_op.cc:270] Start()
0112 20:51:28.171110 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087044399 usec, L: 0
0112 20:51:28.171247 (+ 137us) write_op.cc:270] Start()
0112 20:51:28.171272 (+ 25us) write_op.cc:276] Timestamp: P: 1736715087045321 usec, L: 0
0112 20:51:28.171419 (+ 147us) write_op.cc:270] Start()
0112 20:51:28.171440 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087051322 usec, L: 0
0112 20:51:28.171568 (+ 128us) write_op.cc:270] Start()
0112 20:51:28.171589 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087070831 usec, L: 0
0112 20:51:28.171764 (+ 175us) write_op.cc:270] Start()
0112 20:51:28.171788 (+ 24us) write_op.cc:276] Timestamp: P: 1736715087071504 usec, L: 0
0112 20:51:28.171924 (+ 136us) write_op.cc:270] Start()
0112 20:51:28.171946 (+ 22us) write_op.cc:276] Timestamp: P: 1736715087072345 usec, L: 0
0112 20:51:28.172077 (+ 131us) write_op.cc:270] Start()
0112 20:51:28.172098 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087073164 usec, L: 0
0112 20:51:28.172234 (+ 136us) write_op.cc:270] Start()
0112 20:51:28.172255 (+ 21us) write_op.cc:276] Timestamp: P: 1736715087105747 usec, L: 0
0112 20:51:28.177934 (+ 5679us) log.cc:844] Serialized 110267 byte log entry
0112 20:51:28.178192 (+ 258us) raft_consensus.cc:1640] Marking committed up to 205
0112 20:51:28.184831 (+ 6639us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.184836 (+ 5us) write_op.cc:454] Released schema lock
0112 20:51:28.185497 (+ 661us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.185503 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.186006 (+ 503us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.186012 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.186691 (+ 679us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.186695 (+ 4us) write_op.cc:454] Released schema lock
0112 20:51:28.192839 (+ 6144us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.192845 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.193417 (+ 572us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.193422 (+ 5us) write_op.cc:454] Released schema lock
0112 20:51:28.193897 (+ 475us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.193902 (+ 5us) write_op.cc:454] Released schema lock
0112 20:51:28.194566 (+ 664us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.194573 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:28.195284 (+ 711us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.195291 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:28.195824 (+ 533us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.195831 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:28.196355 (+ 524us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.196361 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.196878 (+ 517us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.196884 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.197387 (+ 503us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.197393 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.197913 (+ 520us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.197919 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.198423 (+ 504us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:28.198429 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:28.199116 (+ 687us) write_op.cc:489] Releasing partition,
I20250112 20:51:28.843299 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 124) took 2165 ms. Trace:
I20250112 20:51:28.843397 1186 rpcz_store.cc:276] 0112 20:51:26.677660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.677727 (+ 67us) service_pool.cc:225] Handling call
0112 20:51:28.843193 (+2165466us) ts_tablet_manager.cc:1145] Deleting tablet 8c38c8594a384689ae57c26174765284
0112 20:51:28.843200 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.843269 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165196}
I20250112 20:51:28.843609 1186 tablet_replica.cc:331] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:28.843804 1186 raft_consensus.cc:2238] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.843972 1186 raft_consensus.cc:2267] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.844585 1186 ts_tablet_manager.cc:1905] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.722244 1164 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:28.845028 1164 ts_tablet_manager.cc:1397] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.425s user 0.088s sys 0.089s
I20250112 20:51:28.845788 1164 raft_consensus.cc:357] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.845981 1164 raft_consensus.cc:383] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.846048 1164 raft_consensus.cc:738] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.846282 1164 consensus_queue.cc:260] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:28.846812 1164 ts_tablet_manager.cc:1428] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:28.847111 1164 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:28.848362 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:28.848618 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:28.719486 1506 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:28.756320 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b985fd656ce44131bc91ccfeb79587c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 206 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
W20250112 20:51:28.726438 847 consensus_peers.cc:487] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.849367 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_558ea8a4fda94f119e0603dcb4ceff52" } modify_external_catalogs: true
W20250112 20:51:28.849460 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 123035us
I20250112 20:51:28.849546 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_558ea8a4fda94f119e0603dcb4ceff52" } modify_external_catalogs: true
W20250112 20:51:28.849766 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ca061b82a894e91b9508613ccd8ecd7 consensus state has no leader
I20250112 20:51:28.850102 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e18e998a23124368951dc9ad456b0567" } modify_external_catalogs: true
I20250112 20:51:28.850239 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e18e998a23124368951dc9ad456b0567" } modify_external_catalogs: true
I20250112 20:51:28.852142 882 tablet_service.cc:1467] Processing CreateTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.852227 889 tablet_service.cc:1467] Processing CreateTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 (DEFAULT_TABLE table=default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.852562 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e390eb2e404250a8b6edba2ee4f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.853001 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 (DEFAULT_TABLE table=default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.853441 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d60604713524dba8f3a42c4f3bc7300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.857079 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 (DEFAULT_TABLE table=default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.857182 879 tablet_service.cc:1467] Processing CreateTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed (DEFAULT_TABLE table=default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2 [id=f7bbf00f03b546d5bbcb872d503aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.857434 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d08048c402345a686d1e3fb3039bbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.857869 881 tablet_service.cc:1467] Processing CreateTablet for tablet a3ca4d49496e42ddb263264ceef2f27a (DEFAULT_TABLE table=default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.858307 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ca4d49496e42ddb263264ceef2f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.860882 876 tablet_service.cc:1467] Processing CreateTablet for tablet 640dc9187c2243248ef73bac6616c126 (DEFAULT_TABLE table=default.Table_4b1f4ea776424da59c8ec69715c5337d [id=e8f4d732342e43bda26b51c8e1122d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.861162 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640dc9187c2243248ef73bac6616c126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.727952 1178 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:28.861956 1178 ts_tablet_manager.cc:1397] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.415s user 0.010s sys 0.002s
W20250112 20:51:28.862178 847 consensus_peers.cc:487] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.863030 1178 raft_consensus.cc:357] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.863287 1178 raft_consensus.cc:383] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.863390 1178 raft_consensus.cc:738] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.863694 1178 consensus_queue.cc:260] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.864359 1178 ts_tablet_manager.cc:1428] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:28.768154 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 12cd493cb197490d87cc7727eec35c72
W20250112 20:51:28.778324 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42eb0f79df5444f8a8360921d5f2b9ee consensus state has no leader
W20250112 20:51:28.779727 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e590f9615c4d47abaad1a36ac74b3c80 consensus state has no leader
I20250112 20:51:28.784669 449 catalog_manager.cc:5526] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:28.864903 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_afbd47897b344f1db9916ab7e703ab36" } modify_external_catalogs: true
I20250112 20:51:28.865135 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_afbd47897b344f1db9916ab7e703ab36" } modify_external_catalogs: true
I20250112 20:51:28.865303 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c7a57ea399194a45bc8990a029550440 (table default.Table_673a2a9243984736b6b43018eb3e1a42 [id=8133cc129a404fb8aca53d1fdb1c7fef]). Expected version 2 got 1
I20250112 20:51:28.740257 1440 raft_consensus.cc:491] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:28.865515 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 4d616a8ad9ad4777b44deb77bc321d66 (table default.Table_afbd47897b344f1db9916ab7e703ab36 [id=2fd5583e6e114bcdbdbc723643735fd8]). Expected version 2 got 0
I20250112 20:51:28.865728 1440 raft_consensus.cc:513] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:28.798017 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e590f9615c4d47abaad1a36ac74b3c80 (table default.Table_4c02bae8e4d64469b992b13dd36c7e81 [id=7dc6e9f691804365a9696db3b26700bd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e590f9615c4d47abaad1a36ac74b3c80 consensus state has no leader
I20250112 20:51:28.755584 1184 ts_tablet_manager.cc:1918] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.866434 1184 log.cc:1198] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/24abf89a9b2542d7bac2f7499698df3a
I20250112 20:51:28.866520 1440 leader_election.cc:290] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:28.866891 1184 ts_tablet_manager.cc:1939] T 24abf89a9b2542d7bac2f7499698df3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:28.867720 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 123) took 2230 ms. Trace:
I20250112 20:51:28.867835 1184 rpcz_store.cc:276] 0112 20:51:26.637345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.639037 (+ 1692us) service_pool.cc:225] Handling call
0112 20:51:28.556396 (+1917359us) ts_tablet_manager.cc:1145] Deleting tablet 24abf89a9b2542d7bac2f7499698df3a
0112 20:51:28.556408 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.736688 (+180280us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.739961 (+ 3273us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.867685 (+127724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1917219}
I20250112 20:51:28.868309 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 125) took 2160 ms. Trace:
I20250112 20:51:28.759487 1163 tablet_bootstrap.cc:492] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:28.868413 1184 rpcz_store.cc:276] 0112 20:51:26.707855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.707967 (+ 112us) service_pool.cc:225] Handling call
0112 20:51:28.868133 (+2160166us) ts_tablet_manager.cc:1145] Deleting tablet 8c38c8594a384689ae57c26174765284
0112 20:51:28.868143 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.868280 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2158132}
I20250112 20:51:28.868594 1163 ts_tablet_manager.cc:1397] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.025s
I20250112 20:51:28.868721 1184 tablet_replica.cc:331] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:28.868954 1184 raft_consensus.cc:2238] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:28.869282 1184 raft_consensus.cc:2267] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.810725 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1dd4ddaed71542489ede444900ece37b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:28.869467 1163 raft_consensus.cc:357] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.869660 1163 raft_consensus.cc:383] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.869772 1163 raft_consensus.cc:738] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:28.869800 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dd4ddaed71542489ede444900ece37b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.870013 1163 consensus_queue.cc:260] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.735128 1185 ts_tablet_manager.cc:1918] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.787861 756 tablet_service.cc:1467] Processing CreateTablet for tablet a3ca4d49496e42ddb263264ceef2f27a (DEFAULT_TABLE table=default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.870307 1185 log.cc:1198] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d48b17628cf145b69e5fdde4131d9fd2
I20250112 20:51:28.870496 1163 ts_tablet_manager.cc:1428] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:28.870625 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ca4d49496e42ddb263264ceef2f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.870729 1185 ts_tablet_manager.cc:1939] T d48b17628cf145b69e5fdde4131d9fd2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:28.870771 1163 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:28.871497 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 137) took 1753 ms. Trace:
I20250112 20:51:28.871584 1185 rpcz_store.cc:276] 0112 20:51:27.117620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.118718 (+ 1098us) service_pool.cc:225] Handling call
0112 20:51:28.512416 (+1393698us) ts_tablet_manager.cc:1145] Deleting tablet d48b17628cf145b69e5fdde4131d9fd2
0112 20:51:28.512429 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.581808 (+ 69379us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.735031 (+153223us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.871436 (+136405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1393495}
I20250112 20:51:28.871958 1185 tablet_replica.cc:331] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:28.872149 1185 raft_consensus.cc:2238] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.872334 1163 tablet_bootstrap.cc:654] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.872331 1185 raft_consensus.cc:2267] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.789055 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed (DEFAULT_TABLE table=default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2 [id=f7bbf00f03b546d5bbcb872d503aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.873090 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c5e95d792c48441db50a9e741756db91" } modify_external_catalogs: true
I20250112 20:51:28.873299 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4303e4aec9694c49aca7c1532eaab3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.873243 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c5e95d792c48441db50a9e741756db91" } modify_external_catalogs: true
I20250112 20:51:28.873340 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 24abf89a9b2542d7bac2f7499698df3a (table default.Table_e7baae11d0fa447dbca4bf5211e3939c [id=c8469f72ecf5453c879fb24e91bdc8a2]) successfully deleted
I20250112 20:51:28.873406 1298 leader_election.cc:304] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b985fd656ce44131bc91ccfeb79587c5; no voters: 0f5c09e899184de6b770dca873501062, b04ae76622ac46f5a584a582e372bc05
W20250112 20:51:28.873584 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8c38c8594a384689ae57c26174765284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c38c8594a384689ae57c26174765284 already in progress: creating tablet
I20250112 20:51:28.874045 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 30e53e67076c4a5a89f1e93fc168a997 (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
I20250112 20:51:28.874183 447 master_service.cc:946] ReplaceTablet: received request to replace tablet d8330fa1e4654a1a9f167a1120a103fc from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.805115 1430 raft_consensus.cc:491] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:28.786368 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a3ca4d49496e42ddb263264ceef2f27a (DEFAULT_TABLE table=default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.879041 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ca4d49496e42ddb263264ceef2f27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.787890 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed (DEFAULT_TABLE table=default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2 [id=f7bbf00f03b546d5bbcb872d503aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.880149 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4303e4aec9694c49aca7c1532eaab3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.789124 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 (DEFAULT_TABLE table=default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.792047 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 (DEFAULT_TABLE table=default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.792352 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 640dc9187c2243248ef73bac6616c126 (DEFAULT_TABLE table=default.Table_4b1f4ea776424da59c8ec69715c5337d [id=e8f4d732342e43bda26b51c8e1122d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.796065 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 (DEFAULT_TABLE table=default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.797677 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.875711 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:28.791203 749 tablet_service.cc:1467] Processing CreateTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 (DEFAULT_TABLE table=default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.794665 751 tablet_service.cc:1467] Processing CreateTablet for tablet 640dc9187c2243248ef73bac6616c126 (DEFAULT_TABLE table=default.Table_4b1f4ea776424da59c8ec69715c5337d [id=e8f4d732342e43bda26b51c8e1122d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.796934 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 (DEFAULT_TABLE table=default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.798174 750 tablet_service.cc:1467] Processing CreateTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.790115 752 tablet_service.cc:1467] Processing CreateTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 (DEFAULT_TABLE table=default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:28.875079 1430 raft_consensus.cc:513] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.891348 1430 leader_election.cc:290] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:28.891861 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d60604713524dba8f3a42c4f3bc7300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.893189 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d8e68746cf444ebac5e22728ab6f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.893945 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d08048c402345a686d1e3fb3039bbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.877678 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4303e4aec9694c49aca7c1532eaab3ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.899225 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e390eb2e404250a8b6edba2ee4f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.900987 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d8e68746cf444ebac5e22728ab6f34. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:28.874258 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 158cfe5ba52c4771af97841885b29f9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158cfe5ba52c4771af97841885b29f9d already in progress: creating tablet
I20250112 20:51:28.874769 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 382aefb4a5e34999b560a9237839b380 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:28.904099 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 7f39195945024f4fba9942faec70bc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f39195945024f4fba9942faec70bc31 already in progress: creating tablet
W20250112 20:51:28.904359 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 29b70ffb3e3341d398d36ff76993e9ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29b70ffb3e3341d398d36ff76993e9ac already in progress: creating tablet
W20250112 20:51:28.904596 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8c38c8594a384689ae57c26174765284 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c38c8594a384689ae57c26174765284 already in progress: creating tablet
I20250112 20:51:28.905088 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d48b17628cf145b69e5fdde4131d9fd2 (table default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]) successfully deleted
I20250112 20:51:28.905720 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aba3e9f239704c2c843595403a218ca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.906318 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aba3e9f239704c2c843595403a218ca7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.878576 1178 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:28.881289 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:28.908979 775 raft_consensus.cc:2463] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:28.908979 1185 ts_tablet_manager.cc:1905] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.909068 1164 tablet_bootstrap.cc:654] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.910857 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640dc9187c2243248ef73bac6616c126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.914244 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet eda44c48bf9746b59ccdc4d39a41ea91
I20250112 20:51:28.918216 876 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:28.921805 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea7336d13ea46748167e35e7e7e8c98" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:28.923534 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e390eb2e404250a8b6edba2ee4f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:28.924274 847 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.924613 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640dc9187c2243248ef73bac6616c126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.925364 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:28.925509 1184 ts_tablet_manager.cc:1905] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.925704 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:28.926538 746 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:28.926914 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d8e68746cf444ebac5e22728ab6f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.927582 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d60604713524dba8f3a42c4f3bc7300. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:28.928488 847 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:28.929013 847 leader_election.cc:304] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:28.929375 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d08048c402345a686d1e3fb3039bbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:28.929677 1430 raft_consensus.cc:2743] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:28.930217 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:28.928817 983 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.934798 1186 ts_tablet_manager.cc:1918] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.935019 1186 log.cc:1198] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8ad0df4285db4bb6954f3e4c8d858115
I20250112 20:51:28.935410 1186 ts_tablet_manager.cc:1939] T 8ad0df4285db4bb6954f3e4c8d858115 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:28.935763 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 125) took 2257 ms. Trace:
I20250112 20:51:28.935889 1186 rpcz_store.cc:276] 0112 20:51:26.678564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.678625 (+ 61us) service_pool.cc:225] Handling call
0112 20:51:28.843561 (+2164936us) ts_tablet_manager.cc:1145] Deleting tablet 8ad0df4285db4bb6954f3e4c8d858115
0112 20:51:28.843568 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.925863 (+ 82295us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.934688 (+ 8825us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.935715 (+ 1027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164630}
I20250112 20:51:28.936426 1186 tablet_replica.cc:331] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:28.936655 1186 raft_consensus.cc:2238] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:28.937268 1186 raft_consensus.cc:2267] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.938077 983 leader_election.cc:304] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:28.939426 1178 tablet_bootstrap.cc:654] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.939515 1440 raft_consensus.cc:2798] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:28.939647 1440 raft_consensus.cc:491] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:28.940450 1440 raft_consensus.cc:3054] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:28.943632 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8ad0df4285db4bb6954f3e4c8d858115 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:28.947213 1440 raft_consensus.cc:513] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.949347 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:28.950047 1186 ts_tablet_manager.cc:1905] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.950094 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:28.950512 770 raft_consensus.cc:3054] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:28.958056 983 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:28.963632 1186 ts_tablet_manager.cc:1918] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:28.963763 1186 log.cc:1198] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/313a5c9905bc445893f1ffa8752ca589
I20250112 20:51:28.964185 1186 ts_tablet_manager.cc:1939] T 313a5c9905bc445893f1ffa8752ca589 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:28.967990 1185 ts_tablet_manager.cc:1918] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.970242 1178 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:28.970481 1178 ts_tablet_manager.cc:1397] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.005s
I20250112 20:51:28.972944 1178 raft_consensus.cc:357] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.973251 1178 raft_consensus.cc:383] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.973368 1178 raft_consensus.cc:738] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:28.973652 1178 consensus_queue.cc:260] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.974691 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_86c60a78bd264bc08c315269fdf3240f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:28.975282 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86c60a78bd264bc08c315269fdf3240f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:28.969323 770 raft_consensus.cc:2463] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:28.975683 1178 ts_tablet_manager.cc:1428] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:28.973937 1185 log.cc:1198] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e7a8468369af418ea4a1b0a681f8ff74
I20250112 20:51:28.976050 1178 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:28.976187 1185 ts_tablet_manager.cc:1939] T e7a8468369af418ea4a1b0a681f8ff74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:28.976598 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e2f2bcbfc0194e0e9342fe49a37abec0" } new_table_name: "default.Table_54cf793124084547a2343f0a71d88796" modify_external_catalogs: true
I20250112 20:51:28.976989 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 138) took 1859 ms. Trace:
I20250112 20:51:28.977105 1185 rpcz_store.cc:276] 0112 20:51:27.117704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.118948 (+ 1244us) service_pool.cc:225] Handling call
0112 20:51:28.871847 (+1752899us) ts_tablet_manager.cc:1145] Deleting tablet e7a8468369af418ea4a1b0a681f8ff74
0112 20:51:28.871856 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.936341 (+ 64485us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.967861 (+ 31520us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.976956 (+ 9095us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1752719}
I20250112 20:51:28.977416 1185 tablet_replica.cc:331] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:28.977594 1185 raft_consensus.cc:2238] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.977800 1185 raft_consensus.cc:2267] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.977869 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b92d295d444f4b398aa900e96f1e0b99" } new_table_name: "default.Table_a5ce6ae642344a2c950ae993e6d90311" modify_external_catalogs: true
I20250112 20:51:28.978454 1185 ts_tablet_manager.cc:1905] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.989346 1178 tablet_bootstrap.cc:654] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:28.964488 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 126) took 2221 ms. Trace:
I20250112 20:51:28.990532 1186 rpcz_store.cc:276] 0112 20:51:26.742952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.743012 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:28.936346 (+2193334us) ts_tablet_manager.cc:1145] Deleting tablet 313a5c9905bc445893f1ffa8752ca589
0112 20:51:28.936356 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.953805 (+ 17449us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.963515 (+ 9710us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.964458 (+ 943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2193095}
I20250112 20:51:28.991132 1186 tablet_replica.cc:331] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:28.991410 1186 raft_consensus.cc:2238] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:28.991673 1186 raft_consensus.cc:2267] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:28.992007 1184 ts_tablet_manager.cc:1918] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:28.992599 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:28.993368 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_673a2a9243984736b6b43018eb3e1a42 [id=8133cc129a404fb8aca53d1fdb1c7fef] alter complete (version 2)
I20250112 20:51:28.994652 1186 ts_tablet_manager.cc:1905] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:28.996030 1185 ts_tablet_manager.cc:1918] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:28.996228 1185 log.cc:1198] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/616b8aa8fd62486c978c275712064f77
I20250112 20:51:28.996639 1185 ts_tablet_manager.cc:1939] T 616b8aa8fd62486c978c275712064f77 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:28.997752 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.001587 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 313a5c9905bc445893f1ffa8752ca589 (table default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]) successfully deleted
I20250112 20:51:29.001950 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.003343 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.004114 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e7a8468369af418ea4a1b0a681f8ff74 (table default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]) successfully deleted
I20250112 20:51:29.002729 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b437c31235f14edb94ca1e295cb9487e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.004972 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b437c31235f14edb94ca1e295cb9487e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.005429 1184 log.cc:1198] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/eb1dffbe43ba4eeab7ee12d901b62ffb
I20250112 20:51:29.007088 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_70dd4d25b3854472992074db0c6fbeb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.007495 1178 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
W20250112 20:51:29.007561 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70dd4d25b3854472992074db0c6fbeb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.007660 1178 ts_tablet_manager.cc:1397] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.005s
I20250112 20:51:29.007930 1184 ts_tablet_manager.cc:1939] T eb1dffbe43ba4eeab7ee12d901b62ffb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.008495 1178 raft_consensus.cc:357] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:28.996940 1440 leader_election.cc:290] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.008805 1178 raft_consensus.cc:383] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:28.997054 983 leader_election.cc:304] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.008982 1178 raft_consensus.cc:738] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.009053 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 126) took 2258 ms. Trace:
I20250112 20:51:29.009197 1184 rpcz_store.cc:276] 0112 20:51:26.750885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.750992 (+ 107us) service_pool.cc:225] Handling call
0112 20:51:28.868656 (+2117664us) ts_tablet_manager.cc:1145] Deleting tablet eb1dffbe43ba4eeab7ee12d901b62ffb
0112 20:51:28.868663 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.957397 (+ 88734us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.991887 (+ 34490us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.008985 (+ 17098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2117420}
I20250112 20:51:29.009238 1178 consensus_queue.cc:260] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.009346 1440 raft_consensus.cc:2798] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:29.009724 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 131) took 2086 ms. Trace:
I20250112 20:51:29.009821 1184 rpcz_store.cc:276] 0112 20:51:26.923323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.923379 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:29.009530 (+2086151us) ts_tablet_manager.cc:1145] Deleting tablet 8ad0df4285db4bb6954f3e4c8d858115
0112 20:51:29.009540 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.009644 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2085904}
I20250112 20:51:29.010565 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 132) took 2001 ms. Trace:
I20250112 20:51:29.010660 1184 rpcz_store.cc:276] 0112 20:51:27.008906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.010466 (+ 1560us) service_pool.cc:225] Handling call
0112 20:51:29.010064 (+1999598us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:29.010070 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.010146 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999311}
I20250112 20:51:29.011018 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 133) took 2001 ms. Trace:
I20250112 20:51:29.011103 1184 rpcz_store.cc:276] 0112 20:51:27.009078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.010799 (+ 1721us) service_pool.cc:225] Handling call
0112 20:51:29.010858 (+2000059us) ts_tablet_manager.cc:1145] Deleting tablet 1c7a54ce78854137ab2d12e9ce055565
0112 20:51:29.010868 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.010973 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999885}
I20250112 20:51:29.011457 1440 raft_consensus.cc:695] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:29.012921 889 tablet_service.cc:1467] Processing CreateTablet for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (DEFAULT_TABLE table=default.Table_dc0bcf302ab44599b97fbba9f0301582 [id=3861236543364f8cbcd9632b17fca2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.013474 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.014201 751 tablet_service.cc:1467] Processing CreateTablet for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (DEFAULT_TABLE table=default.Table_dc0bcf302ab44599b97fbba9f0301582 [id=3861236543364f8cbcd9632b17fca2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.014678 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.024284 879 tablet_service.cc:1467] Processing CreateTablet for tablet 85897bfec1834c57956a23c5dfc2e853 (DEFAULT_TABLE table=default.Table_66f53100719d424aa91c8c1ddad82c8a [id=1de4457277634138946b0e0710d31cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.024573 1164 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.024734 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85897bfec1834c57956a23c5dfc2e853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.024715 1164 ts_tablet_manager.cc:1397] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.178s user 0.005s sys 0.009s
I20250112 20:51:29.025799 1164 raft_consensus.cc:357] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.026018 1164 raft_consensus.cc:383] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.026101 1164 raft_consensus.cc:738] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.026310 1164 consensus_queue.cc:260] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.028265 1164 ts_tablet_manager.cc:1428] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:29.028539 1164 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.032306 1164 tablet_bootstrap.cc:654] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.033257 1178 ts_tablet_manager.cc:1428] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
W20250112 20:51:29.034698 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7293510f40804e2aabccf92986489dab consensus state has no leader
I20250112 20:51:29.034983 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 158cfe5ba52c4771af97841885b29f9d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.036446 1178 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.036624 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:29.036938 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 4390c60d6c724190a216ef9c961b9592 (table default.Table_9c771feed27441ec8340f7279a4a06c1 [id=2ee1f97fef0f4e2b83932132d0936504]). Expected version 2 got 0
I20250112 20:51:29.038024 752 tablet_service.cc:1467] Processing CreateTablet for tablet 85897bfec1834c57956a23c5dfc2e853 (DEFAULT_TABLE table=default.Table_66f53100719d424aa91c8c1ddad82c8a [id=1de4457277634138946b0e0710d31cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.038479 1178 tablet_bootstrap.cc:654] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.038507 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85897bfec1834c57956a23c5dfc2e853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.041180 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f0092258e094a38ba7e223be03d4c9a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:28.997773 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 146) took 1850 ms. Trace:
I20250112 20:51:29.042140 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3b532068d2374b59a3c46525db7c50db (DEFAULT_TABLE table=default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.042491 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b532068d2374b59a3c46525db7c50db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.042464 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet eb1dffbe43ba4eeab7ee12d901b62ffb (table default.Table_20471004d83e485dbea6520c7bdb37b0 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
W20250112 20:51:29.042721 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8ad0df4285db4bb6954f3e4c8d858115 because the tablet was not found. No further retry: Not found: Tablet not found: 8ad0df4285db4bb6954f3e4c8d858115
W20250112 20:51:29.042963 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 304e1c8abe094bd99a99fd358b697f81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 304e1c8abe094bd99a99fd358b697f81 already in progress: creating tablet
I20250112 20:51:29.039144 879 tablet_service.cc:1467] Processing CreateTablet for tablet 3b532068d2374b59a3c46525db7c50db (DEFAULT_TABLE table=default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.044409 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b532068d2374b59a3c46525db7c50db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.044664 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 3b532068d2374b59a3c46525db7c50db (DEFAULT_TABLE table=default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.045121 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b532068d2374b59a3c46525db7c50db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.048584 753 tablet_service.cc:1467] Processing CreateTablet for tablet 06253bb13a9541d8b408d1bf73e469dd (DEFAULT_TABLE table=default.Table_47f89b433c7e4b11a6e685b3c702c3d1 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.049010 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06253bb13a9541d8b408d1bf73e469dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.049048 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 06253bb13a9541d8b408d1bf73e469dd (DEFAULT_TABLE table=default.Table_47f89b433c7e4b11a6e685b3c702c3d1 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.049378 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06253bb13a9541d8b408d1bf73e469dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.053977 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 85897bfec1834c57956a23c5dfc2e853 (DEFAULT_TABLE table=default.Table_66f53100719d424aa91c8c1ddad82c8a [id=1de4457277634138946b0e0710d31cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.054342 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85897bfec1834c57956a23c5dfc2e853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.060431 1178 tablet_bootstrap.cc:492] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.060658 1178 ts_tablet_manager.cc:1397] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20250112 20:51:29.061470 1178 raft_consensus.cc:357] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.061815 1178 raft_consensus.cc:383] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.061962 1178 raft_consensus.cc:738] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.062320 1178 consensus_queue.cc:260] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.062765 751 tablet_service.cc:1467] Processing CreateTablet for tablet 38cdb9a415f745cbb62f27763e490d5b (DEFAULT_TABLE table=default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e [id=d20f36b81b8d4a1ba8c0c792e1a997c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.063210 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38cdb9a415f745cbb62f27763e490d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.064466 1178 ts_tablet_manager.cc:1428] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:29.064865 1178 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.067379 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a218ef5825424b1b84c55b38b9b9023d (DEFAULT_TABLE table=default.Table_40c44749c3934a83ab1b8ed501bf1c91 [id=05211d8527144710860dc2aca1ac76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.067620 1178 tablet_bootstrap.cc:654] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.068173 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a218ef5825424b1b84c55b38b9b9023d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.063334 1185 rpcz_store.cc:276] 0112 20:51:27.147546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.147656 (+ 110us) service_pool.cc:225] Handling call
0112 20:51:28.977358 (+1829702us) ts_tablet_manager.cc:1145] Deleting tablet 616b8aa8fd62486c978c275712064f77
0112 20:51:28.977368 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:28.985817 (+ 8449us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.995922 (+ 10105us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:28.997712 (+ 1790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829483}
I20250112 20:51:29.069365 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 147) took 1839 ms. Trace:
I20250112 20:51:29.069521 1185 rpcz_store.cc:276] 0112 20:51:27.229584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.229649 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:29.069199 (+1839550us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:29.069208 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.069316 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1839352}
I20250112 20:51:29.070040 1185 tablet_replica.cc:331] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.011865 1440 consensus_queue.cc:237] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.070304 1185 raft_consensus.cc:2238] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.070405 1441 raft_consensus.cc:491] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.070526 1441 raft_consensus.cc:513] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.070686 1185 raft_consensus.cc:2267] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.071245 1185 ts_tablet_manager.cc:1905] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.071861 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:29.072649 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc63526f932841509277f4726303c801" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:29.040066 889 tablet_service.cc:1467] Processing CreateTablet for tablet 06253bb13a9541d8b408d1bf73e469dd (DEFAULT_TABLE table=default.Table_47f89b433c7e4b11a6e685b3c702c3d1 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.041268 882 tablet_service.cc:1467] Processing CreateTablet for tablet 38cdb9a415f745cbb62f27763e490d5b (DEFAULT_TABLE table=default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e [id=d20f36b81b8d4a1ba8c0c792e1a997c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.076249 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38cdb9a415f745cbb62f27763e490d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.082500 1164 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.082712 1164 ts_tablet_manager.cc:1397] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.002s
I20250112 20:51:29.083575 1164 raft_consensus.cc:357] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.042577 1186 ts_tablet_manager.cc:1918] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:29.083851 1164 raft_consensus.cc:383] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.083988 1186 log.cc:1198] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/eb1dffbe43ba4eeab7ee12d901b62ffb
I20250112 20:51:29.083995 1164 raft_consensus.cc:738] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.084408 1186 ts_tablet_manager.cc:1939] T eb1dffbe43ba4eeab7ee12d901b62ffb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.084334 1164 consensus_queue.cc:260] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.085503 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 127) took 2335 ms. Trace:
I20250112 20:51:29.086134 1164 ts_tablet_manager.cc:1428] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:29.086493 1164 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.085603 1186 rpcz_store.cc:276] 0112 20:51:26.749779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.750320 (+ 541us) service_pool.cc:225] Handling call
0112 20:51:28.991060 (+2240740us) ts_tablet_manager.cc:1145] Deleting tablet eb1dffbe43ba4eeab7ee12d901b62ffb
0112 20:51:28.991070 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.000856 (+ 9786us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.042467 (+ 41611us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.085448 (+ 42981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2240558}
I20250112 20:51:29.087428 1186 tablet_replica.cc:331] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.087710 1186 raft_consensus.cc:2238] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.087960 1186 raft_consensus.cc:2267] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.089874 1164 tablet_bootstrap.cc:654] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.089864 1186 ts_tablet_manager.cc:1905] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.091215 1185 ts_tablet_manager.cc:1918] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:29.091940 1185 log.cc:1198] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e040383f6e2a4ccf9614fd2d119a3cb4
I20250112 20:51:29.092420 1185 ts_tablet_manager.cc:1939] T e040383f6e2a4ccf9614fd2d119a3cb4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.093133 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06253bb13a9541d8b408d1bf73e469dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.093343 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 148) took 1814 ms. Trace:
I20250112 20:51:29.093495 1185 rpcz_store.cc:276] 0112 20:51:27.279093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.280504 (+ 1411us) service_pool.cc:225] Handling call
0112 20:51:29.069967 (+1789463us) ts_tablet_manager.cc:1145] Deleting tablet e040383f6e2a4ccf9614fd2d119a3cb4
0112 20:51:29.069975 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.074197 (+ 4222us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.091087 (+ 16890us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.093309 (+ 2222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723555}
I20250112 20:51:29.094139 1185 tablet_replica.cc:331] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.094386 1185 raft_consensus.cc:2238] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.094600 1185 raft_consensus.cc:2267] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.095161 1185 ts_tablet_manager.cc:1905] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.105895 983 leader_election.cc:304] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.106752 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 38cdb9a415f745cbb62f27763e490d5b (DEFAULT_TABLE table=default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e [id=d20f36b81b8d4a1ba8c0c792e1a997c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.107122 1185 ts_tablet_manager.cc:1918] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.107457 1185 log.cc:1198] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9a692cc5fb6f4afc854c4821274edfdb
I20250112 20:51:29.107506 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38cdb9a415f745cbb62f27763e490d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.108073 1185 ts_tablet_manager.cc:1939] T 9a692cc5fb6f4afc854c4821274edfdb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.109954 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 149) took 1828 ms. Trace:
I20250112 20:51:29.110466 1185 rpcz_store.cc:276] 0112 20:51:27.281314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.282983 (+ 1669us) service_pool.cc:225] Handling call
0112 20:51:29.094082 (+1811099us) ts_tablet_manager.cc:1145] Deleting tablet 9a692cc5fb6f4afc854c4821274edfdb
0112 20:51:29.094091 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.100607 (+ 6516us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.106001 (+ 5394us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.109897 (+ 3896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1743514}
I20250112 20:51:29.111114 1354 raft_consensus.cc:2743] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.111493 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 164) took 1750 ms. Trace:
I20250112 20:51:29.111835 1441 leader_election.cc:290] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.111857 1185 rpcz_store.cc:276] 0112 20:51:27.361065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.361110 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:29.111187 (+1750077us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:29.111336 (+ 149us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.111443 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749878}
I20250112 20:51:29.112112 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (DEFAULT_TABLE table=default.Table_dc0bcf302ab44599b97fbba9f0301582 [id=3861236543364f8cbcd9632b17fca2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.112650 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.112957 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 166) took 1746 ms. Trace:
I20250112 20:51:29.113160 1185 rpcz_store.cc:276] 0112 20:51:27.366455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.366561 (+ 106us) service_pool.cc:225] Handling call
0112 20:51:29.112575 (+1746014us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:29.112584 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.112669 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740390}
I20250112 20:51:29.113876 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 165) took 1747 ms. Trace:
I20250112 20:51:29.114300 1185 rpcz_store.cc:276] 0112 20:51:27.365966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.366186 (+ 220us) service_pool.cc:225] Handling call
0112 20:51:29.113749 (+1747563us) ts_tablet_manager.cc:1145] Deleting tablet af1aac31fd32400b9094f8e2de4fe42c
0112 20:51:29.113757 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.113829 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738874}
I20250112 20:51:29.115025 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 177) took 1616 ms. Trace:
I20250112 20:51:29.115422 1185 rpcz_store.cc:276] 0112 20:51:27.498140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.503748 (+ 5608us) service_pool.cc:225] Handling call
0112 20:51:29.114881 (+1611133us) ts_tablet_manager.cc:1145] Deleting tablet 6fe28f23448c47d3979f1982f939e39c
0112 20:51:29.114889 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.114980 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1610899}
I20250112 20:51:29.116160 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 179) took 1518 ms. Trace:
I20250112 20:51:29.116555 1185 rpcz_store.cc:276] 0112 20:51:27.597998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.605043 (+ 7045us) service_pool.cc:225] Handling call
0112 20:51:29.116023 (+1510980us) ts_tablet_manager.cc:1145] Deleting tablet 0e2235e2a63c4e1192b3260085908445
0112 20:51:29.116032 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.116109 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510293}
I20250112 20:51:29.117259 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 180) took 1442 ms. Trace:
I20250112 20:51:29.117630 1185 rpcz_store.cc:276] 0112 20:51:27.674956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.675005 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:29.117136 (+1442131us) ts_tablet_manager.cc:1145] Deleting tablet b84624dc685b401192ca753ff59da016
0112 20:51:29.117145 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.117213 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441878}
I20250112 20:51:29.118324 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 181) took 1442 ms. Trace:
I20250112 20:51:29.118748 1185 rpcz_store.cc:276] 0112 20:51:27.676171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.676228 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:29.118187 (+1441959us) ts_tablet_manager.cc:1145] Deleting tablet 8b414be57990432bb786daf5faf0cc2e
0112 20:51:29.118195 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.118277 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441705}
I20250112 20:51:29.119441 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 182) took 1441 ms. Trace:
I20250112 20:51:29.119877 1185 rpcz_store.cc:276] 0112 20:51:27.677839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.677931 (+ 92us) service_pool.cc:225] Handling call
0112 20:51:29.119304 (+1441373us) ts_tablet_manager.cc:1145] Deleting tablet d8330fa1e4654a1a9f167a1120a103fc
0112 20:51:29.119312 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.119386 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441182}
I20250112 20:51:29.120996 1185 tablet_replica.cc:331] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.121119 1163 tablet_bootstrap.cc:492] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.121433 1185 raft_consensus.cc:2238] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.121887 1185 raft_consensus.cc:2267] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.122644 1185 ts_tablet_manager.cc:1905] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.123597 1163 ts_tablet_manager.cc:1397] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.253s user 0.005s sys 0.010s
I20250112 20:51:29.124522 1163 raft_consensus.cc:357] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:29.129868 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 86883.7us
I20250112 20:51:29.063187 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3c14102aecb743e2a82660039bd0f038
I20250112 20:51:29.134333 1185 ts_tablet_manager.cc:1918] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.134457 1185 log.cc:1198] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/0871d4272e874a958174f96bfc0e1a95
I20250112 20:51:29.134562 1178 tablet_bootstrap.cc:492] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.134799 1178 ts_tablet_manager.cc:1397] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.000s
W20250112 20:51:29.134799 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1c7a54ce78854137ab2d12e9ce055565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c7a54ce78854137ab2d12e9ce055565 already in progress: creating tablet
I20250112 20:51:29.135144 1163 raft_consensus.cc:383] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.135491 1163 raft_consensus.cc:738] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.135617 1178 raft_consensus.cc:357] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.135913 1178 raft_consensus.cc:383] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.136040 1178 raft_consensus.cc:738] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.136313 1178 consensus_queue.cc:260] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.137266 1185 ts_tablet_manager.cc:1939] T 0871d4272e874a958174f96bfc0e1a95 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.069841 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
I20250112 20:51:29.071542 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:29.139307 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:29.139631 882 tablet_service.cc:1467] Processing CreateTablet for tablet a218ef5825424b1b84c55b38b9b9023d (DEFAULT_TABLE table=default.Table_40c44749c3934a83ab1b8ed501bf1c91 [id=05211d8527144710860dc2aca1ac76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.139854 1178 ts_tablet_manager.cc:1428] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:29.140039 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a218ef5825424b1b84c55b38b9b9023d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.141453 1185 tablet_replica.cc:331] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.141659 1185 raft_consensus.cc:2238] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.141808 1185 raft_consensus.cc:2267] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.142336 1185 ts_tablet_manager.cc:1905] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.143424 1178 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.072688 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 572172a274494e429b7e982815897e61 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.145076 1163 consensus_queue.cc:260] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.129534 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 134) took 2104 ms. Trace:
I20250112 20:51:29.074496 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:29.145368 1184 rpcz_store.cc:276] 0112 20:51:27.024708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.024820 (+ 112us) service_pool.cc:225] Handling call
0112 20:51:29.011337 (+1986517us) ts_tablet_manager.cc:1145] Deleting tablet 3ade9b410d8443249730cacdf7266eb2
0112 20:51:29.011347 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.129469 (+118122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1986195}
I20250112 20:51:29.145642 1164 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.145777 1164 ts_tablet_manager.cc:1397] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.008s
I20250112 20:51:29.146001 1178 tablet_bootstrap.cc:654] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.145973 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 135) took 2110 ms. Trace:
I20250112 20:51:29.146647 1164 raft_consensus.cc:357] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.146852 1164 raft_consensus.cc:383] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.146956 1164 raft_consensus.cc:738] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.076323 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d184983b685e4a4cb571a72c242ef9b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.147186 1164 consensus_queue.cc:260] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:29.147491 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d184983b685e4a4cb571a72c242ef9b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.077788 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8399a4497e68459fb8e495749b55f0a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.148953 1164 ts_tablet_manager.cc:1428] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250112 20:51:29.149120 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8399a4497e68459fb8e495749b55f0a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.149219 1164 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.081605 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:29.151779 453 catalog_manager.cc:6596] ReplaceTablet: tablet 9f0092258e094a38ba7e223be03d4c9a deleted and replaced by tablet 4bd46a422093495ba8b5b03039874a32
I20250112 20:51:29.152081 1164 tablet_bootstrap.cc:654] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.152411 1451 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 20:51:29.046630 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e72fda1743fc4f96b29add6e36e884de consensus state has no leader
I20250112 20:51:29.065639 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4d616a8ad9ad4777b44deb77bc321d66
I20250112 20:51:29.153033 1335 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" instance_seqno: 1736715083039354) as {username='slave'} at 127.0.104.2:57331; Asking this server to re-register.
I20250112 20:51:29.153318 1332 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" instance_seqno: 1736715082556960) as {username='slave'} at 127.0.104.1:55501; Asking this server to re-register.
I20250112 20:51:29.154182 960 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:29.156478 1334 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" instance_seqno: 1736715083518275) as {username='slave'} at 127.0.104.3:44261; Asking this server to re-register.
I20250112 20:51:29.157022 1185 ts_tablet_manager.cc:1918] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.157212 1328 ts_manager.cc:194] Registered new tserver with Master: 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.157778 823 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:29.159920 1096 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:29.160562 1185 log.cc:1198] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:29.161211 1185 ts_tablet_manager.cc:1939] T e0f1c76db0d64624bd43f0e538f0649d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.163153 1328 ts_manager.cc:194] Registered new tserver with Master: 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.164714 1185 tablet_replica.cc:331] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.165372 1185 raft_consensus.cc:2238] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.165805 1185 raft_consensus.cc:2267] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.166429 1163 ts_tablet_manager.cc:1428] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20250112 20:51:29.166853 1163 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.167025 1185 ts_tablet_manager.cc:1905] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.166487 1164 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.167295 1164 ts_tablet_manager.cc:1397] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20250112 20:51:29.168161 1164 raft_consensus.cc:357] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.168510 1164 raft_consensus.cc:383] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.168622 1164 raft_consensus.cc:738] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.168953 1164 consensus_queue.cc:260] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.169322 1163 tablet_bootstrap.cc:654] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.170547 1164 ts_tablet_manager.cc:1428] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:29.170889 1164 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.171386 1334 ts_manager.cc:194] Registered new tserver with Master: aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.176769 1164 tablet_bootstrap.cc:654] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.177929 751 tablet_service.cc:1467] Processing CreateTablet for tablet a218ef5825424b1b84c55b38b9b9023d (DEFAULT_TABLE table=default.Table_40c44749c3934a83ab1b8ed501bf1c91 [id=05211d8527144710860dc2aca1ac76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.178328 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a218ef5825424b1b84c55b38b9b9023d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.178939 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (DEFAULT_TABLE table=default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.179309 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0a808d431a43fb9299fec1dcf6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.190483 1185 ts_tablet_manager.cc:1918] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.192072 1185 log.cc:1198] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:29.192675 1185 ts_tablet_manager.cc:1939] T 29b70ffb3e3341d398d36ff76993e9ac P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.193445 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
I20250112 20:51:29.196651 1164 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.196867 1164 ts_tablet_manager.cc:1397] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20250112 20:51:29.198015 1164 raft_consensus.cc:357] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.198289 1164 raft_consensus.cc:383] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.198421 1164 raft_consensus.cc:738] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.198642 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 572172a274494e429b7e982815897e61
I20250112 20:51:29.198709 1164 consensus_queue.cc:260] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.199105 454 catalog_manager.cc:6596] ReplaceTablet: tablet 572172a274494e429b7e982815897e61 deleted and replaced by tablet 5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:29.202070 1164 ts_tablet_manager.cc:1428] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:29.202462 1164 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.204808 1164 tablet_bootstrap.cc:654] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.208439 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (DEFAULT_TABLE table=default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.208685 1427 raft_consensus.cc:491] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.208863 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0a808d431a43fb9299fec1dcf6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.208930 1427 raft_consensus.cc:513] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.209545 1427 leader_election.cc:290] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.210378 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d3f3e597f4876bce5852961f37ea0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.211045 710 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.220726 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.221269 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (DEFAULT_TABLE table=default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.221781 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7f79b3309e45afb9e18cad97ddfeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.225211 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (DEFAULT_TABLE table=default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.225796 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7f79b3309e45afb9e18cad97ddfeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.233950 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.235241 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_144f2be878e14cb2901b6993ebee9191"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.236069 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_144f2be878e14cb2901b6993ebee9191 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.236562 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 616b8aa8fd62486c978c275712064f77 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
I20250112 20:51:29.237768 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d83a3643f7974d1ab7d542bdc1dea5ea" } new_table_name: "default.Table_eee58a17649b4263951fabefc0a4dfbf" modify_external_catalogs: true
I20250112 20:51:29.239338 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_983423b832f34c34a969cb2ad8f6db40" } new_table_name: "default.Table_5cf59aa9a65542c1b2bab56c2238c6a7" modify_external_catalogs: true
I20250112 20:51:29.178196 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (DEFAULT_TABLE table=default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.240845 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ac6a910be76848cf9870b52ef755a7b9" } new_table_name: "default.Table_361fe8fc7b6845369a0ee99bc93d2771" modify_external_catalogs: true
I20250112 20:51:29.241293 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0a808d431a43fb9299fec1dcf6d3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.242290 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_30306a177be64cc2b4c2a1ed3351b4de" } modify_external_catalogs: true
I20250112 20:51:29.242480 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_30306a177be64cc2b4c2a1ed3351b4de" } modify_external_catalogs: true
I20250112 20:51:29.243175 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_342a368a27a14267af730e53365d41d1" } modify_external_catalogs: true
I20250112 20:51:29.243364 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_342a368a27a14267af730e53365d41d1" } modify_external_catalogs: true
I20250112 20:51:29.243996 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_90937bd663d44575baa256f844890ba9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.244644 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90937bd663d44575baa256f844890ba9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.245107 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_15613426e60a4f298722927f651c546d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.245652 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15613426e60a4f298722927f651c546d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.246881 449 master_service.cc:946] ReplaceTablet: received request to replace tablet fc63526f932841509277f4726303c801 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.248222 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 8c38c8594a384689ae57c26174765284 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.248997 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.249563 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.189747 1184 rpcz_store.cc:276] 0112 20:51:27.035184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.036068 (+ 884us) service_pool.cc:225] Handling call
0112 20:51:29.145788 (+2109720us) ts_tablet_manager.cc:1145] Deleting tablet 953d2ec8a7564c0bbcf0166487dd1746
0112 20:51:29.145800 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.145896 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109466}
I20250112 20:51:29.211652 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d3f3e597f4876bce5852961f37ea0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.218147 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 719b906973a8490898d6befce49910d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 271b8776c774488089ab1c794dbf823e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.218636 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (DEFAULT_TABLE table=default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.250303 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7f79b3309e45afb9e18cad97ddfeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.250548 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 136) took 2211 ms. Trace:
I20250112 20:51:29.250639 1184 rpcz_store.cc:276] 0112 20:51:27.038662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.038719 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:29.250375 (+2211656us) ts_tablet_manager.cc:1145] Deleting tablet ed45eb5cbd2e489c83c3d343ca1adcfc
0112 20:51:29.250383 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.250498 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211385}
I20250112 20:51:29.251255 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 137) took 2209 ms. Trace:
I20250112 20:51:29.251365 1184 rpcz_store.cc:276] 0112 20:51:27.041867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.042724 (+ 857us) service_pool.cc:225] Handling call
0112 20:51:29.251026 (+2208302us) ts_tablet_manager.cc:1145] Deleting tablet 296703eedc6d459880a74b38e642bceb
0112 20:51:29.251039 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.251130 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208072}
I20250112 20:51:29.251845 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 138) took 2152 ms. Trace:
I20250112 20:51:29.251938 1184 rpcz_store.cc:276] 0112 20:51:27.099391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.099472 (+ 81us) service_pool.cc:225] Handling call
0112 20:51:29.251691 (+2152219us) ts_tablet_manager.cc:1145] Deleting tablet b32dc02497f64a2e9ad99da6c9115d5d
0112 20:51:29.251701 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.251776 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2151919}
I20250112 20:51:29.220144 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.252861 1184 tablet_replica.cc:331] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:29.253050 1184 raft_consensus.cc:2238] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:29.253068 710 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.253192 710 leader_election.cc:304] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.253244 1184 raft_consensus.cc:2267] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.253834 1427 raft_consensus.cc:2743] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.254290 1184 ts_tablet_manager.cc:1905] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:29.257840 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 00c0712ab2de43eea8c77ab39a971a2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c0712ab2de43eea8c77ab39a971a2c already in progress: creating tablet
I20250112 20:51:29.258430 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]) successfully deleted
I20250112 20:51:29.258790 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9a692cc5fb6f4afc854c4821274edfdb (table default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]) successfully deleted
W20250112 20:51:29.259958 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bc5af0bc83bc415b972cbbe0fb39f75f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc5af0bc83bc415b972cbbe0fb39f75f already in progress: creating tablet
W20250112 20:51:29.260368 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f769533bfde0448792d01b883e41763c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f769533bfde0448792d01b883e41763c already in progress: creating tablet
W20250112 20:51:29.260679 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet af1aac31fd32400b9094f8e2de4fe42c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af1aac31fd32400b9094f8e2de4fe42c already in progress: creating tablet
W20250112 20:51:29.261096 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6fe28f23448c47d3979f1982f939e39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fe28f23448c47d3979f1982f939e39c already in progress: creating tablet
W20250112 20:51:29.261422 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 0e2235e2a63c4e1192b3260085908445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2235e2a63c4e1192b3260085908445 already in progress: creating tablet
W20250112 20:51:29.261718 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b84624dc685b401192ca753ff59da016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84624dc685b401192ca753ff59da016 already in progress: creating tablet
W20250112 20:51:29.262051 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8b414be57990432bb786daf5faf0cc2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b414be57990432bb786daf5faf0cc2e already in progress: creating tablet
I20250112 20:51:29.272529 1163 tablet_bootstrap.cc:492] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.272869 1163 ts_tablet_manager.cc:1397] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.001s
I20250112 20:51:29.273069 1440 raft_consensus.cc:491] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.273226 1440 raft_consensus.cc:513] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.273900 1440 leader_election.cc:290] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.273783 1163 raft_consensus.cc:357] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.274557 1163 raft_consensus.cc:383] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.274603 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:29.274693 1163 raft_consensus.cc:738] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.274821 770 raft_consensus.cc:2463] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:29.274921 1163 consensus_queue.cc:260] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.275586 1163 ts_tablet_manager.cc:1428] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:29.276031 1163 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.278098 1163 tablet_bootstrap.cc:654] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.278119 983 leader_election.cc:304] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:29.279000 1440 raft_consensus.cc:2798] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:29.279127 1440 raft_consensus.cc:491] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:29.279214 1440 raft_consensus.cc:3054] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.284063 1440 raft_consensus.cc:513] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.284266 1163 tablet_bootstrap.cc:492] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.284927 1163 ts_tablet_manager.cc:1397] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20250112 20:51:29.285867 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:29.286103 770 raft_consensus.cc:3054] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.285986 1163 raft_consensus.cc:357] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.287284 1163 raft_consensus.cc:383] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.287825 1163 raft_consensus.cc:738] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.288295 1163 consensus_queue.cc:260] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.288841 1440 leader_election.cc:290] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.289397 1163 ts_tablet_manager.cc:1428] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:29.289657 770 raft_consensus.cc:2463] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:29.290175 1163 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.290202 983 leader_election.cc:304] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:29.291035 1440 raft_consensus.cc:2798] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:29.291292 1440 raft_consensus.cc:695] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:29.291719 1440 consensus_queue.cc:237] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:29.292897 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d8330fa1e4654a1a9f167a1120a103fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8330fa1e4654a1a9f167a1120a103fc already in progress: creating tablet
W20250112 20:51:29.293298 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cc971026f6b44db6af8215890ba71075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc971026f6b44db6af8215890ba71075 already in progress: creating tablet
W20250112 20:51:29.294799 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ab5ecd2698b4952a892135b3593106a consensus state has no leader
W20250112 20:51:29.295382 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81f0875e271d4aaf9dc5da6aa8113ed0 consensus state has no leader
W20250112 20:51:29.294483 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4cae067df1a40098f81cfbd17224c75 consensus state has no leader
W20250112 20:51:29.296432 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 201a7e2f17bb46fa918dad8f878a4065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 201a7e2f17bb46fa918dad8f878a4065 already in progress: creating tablet
W20250112 20:51:29.297366 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6f4421d16e3f423794c44970a2469b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4421d16e3f423794c44970a2469b10 already in progress: creating tablet
I20250112 20:51:29.297633 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e0f1c76db0d64624bd43f0e538f0649d (table default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981 [id=5178ded07b844cf4abc1616c387b8585]) successfully deleted
I20250112 20:51:29.300621 1163 tablet_bootstrap.cc:654] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:29.300976 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f249d70b116443b7823e093efeda5cc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f249d70b116443b7823e093efeda5cc3 already in progress: creating tablet
W20250112 20:51:29.303189 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet eda44c48bf9746b59ccdc4d39a41ea91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eda44c48bf9746b59ccdc4d39a41ea91 already in progress: creating tablet
I20250112 20:51:29.304828 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet fc63526f932841509277f4726303c801
I20250112 20:51:29.306142 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 29b70ffb3e3341d398d36ff76993e9ac (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
W20250112 20:51:29.306731 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3c14102aecb743e2a82660039bd0f038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c14102aecb743e2a82660039bd0f038 already in progress: creating tablet
I20250112 20:51:29.306726 449 catalog_manager.cc:6596] ReplaceTablet: tablet fc63526f932841509277f4726303c801 deleted and replaced by tablet d0829e9dd5854961b5402b5710624612
I20250112 20:51:29.309301 1164 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.310756 1164 ts_tablet_manager.cc:1397] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.108s user 0.031s sys 0.033s
I20250112 20:51:29.313899 1164 raft_consensus.cc:357] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.316733 1164 raft_consensus.cc:383] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.317268 1164 raft_consensus.cc:738] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.317836 1164 consensus_queue.cc:260] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.319523 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet eb1dffbe43ba4eeab7ee12d901b62ffb (table default.Table_20471004d83e485dbea6520c7bdb37b0 [id=1d1adf4313fb4c69803023132dd314c2]) successfully deleted
I20250112 20:51:29.322649 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.323016 1185 tablet_replica.cc:331] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.323347 1185 raft_consensus.cc:2238] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.323511 1185 raft_consensus.cc:2267] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.324075 1185 ts_tablet_manager.cc:1905] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.324384 1184 ts_tablet_manager.cc:1918] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:29.326980 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.324525 1184 log.cc:1198] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/313a5c9905bc445893f1ffa8752ca589
I20250112 20:51:29.327695 1184 ts_tablet_manager.cc:1939] T 313a5c9905bc445893f1ffa8752ca589 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.328037 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 23d5269a5c8c4c049265e18bff4e2c45
I20250112 20:51:29.328490 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 146) took 2127 ms. Trace:
I20250112 20:51:29.328595 1184 rpcz_store.cc:276] 0112 20:51:27.201033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.225323 (+ 24290us) service_pool.cc:225] Handling call
0112 20:51:29.252260 (+2026937us) ts_tablet_manager.cc:1145] Deleting tablet 313a5c9905bc445893f1ffa8752ca589
0112 20:51:29.252269 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.267247 (+ 14978us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.318953 (+ 51706us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.328451 (+ 9498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2025873}
I20250112 20:51:29.330421 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.330726 1185 ts_tablet_manager.cc:1918] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.330862 1185 log.cc:1198] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:29.331198 1185 ts_tablet_manager.cc:1939] T b13d3f3e597f4876bce5852961f37ea0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.332927 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.332957 1164 ts_tablet_manager.cc:1428] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250112 20:51:29.335557 1440 raft_consensus.cc:491] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.335798 1440 raft_consensus.cc:513] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.337891 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5d384628ba9b4f81b768eda4f796632b
W20250112 20:51:29.337903 983 leader_election.cc:343] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e7a8468369af418ea4a1b0a681f8ff74
I20250112 20:51:29.338202 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.338459 1440 leader_election.cc:290] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.338820 1164 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.340484 1164 tablet_bootstrap.cc:654] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.340973 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.341210 1185 tablet_replica.cc:331] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.341624 666 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:29.341826 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.342103 1185 raft_consensus.cc:2238] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.342311 1185 raft_consensus.cc:2267] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.343001 1185 ts_tablet_manager.cc:1905] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.347137 1430 raft_consensus.cc:491] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.347379 1430 raft_consensus.cc:513] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.349355 1430 leader_election.cc:290] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.349422 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:29.349907 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1a04c223889f424e831e82a52a84f804 (DEFAULT_TABLE table=default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.350402 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a04c223889f424e831e82a52a84f804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.350458 1164 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.350586 1164 ts_tablet_manager.cc:1397] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20250112 20:51:29.351032 1427 raft_consensus.cc:491] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.351162 1427 raft_consensus.cc:513] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.351740 1427 leader_election.cc:290] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.352622 911 raft_consensus.cc:2463] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:29.353122 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed45eb5cbd2e489c83c3d343ca1adcfc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.353523 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.354100 710 leader_election.cc:343] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.354923 1436 raft_consensus.cc:491] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.355057 1436 raft_consensus.cc:513] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.356133 1436 leader_election.cc:290] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.358325 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84ab3a8fc4580b971d6b9adb18d6f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:29.359123 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1a04c223889f424e831e82a52a84f804 (DEFAULT_TABLE table=default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.361647 913 raft_consensus.cc:3054] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.361753 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c79ce3118cbf49e287c93a76398521da (DEFAULT_TABLE table=default.Table_1dd4ddaed71542489ede444900ece37b [id=c7f63522332840b6bc6cd7462015ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.362790 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79ce3118cbf49e287c93a76398521da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.364888 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a8468369af418ea4a1b0a681f8ff74" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:29.365255 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.365712 710 leader_election.cc:343] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.365871 710 leader_election.cc:304] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.366358 1427 raft_consensus.cc:2743] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.366581 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.367020 983 leader_election.cc:304] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.368223 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 147) took 2138 ms. Trace:
I20250112 20:51:29.368315 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.364750 913 raft_consensus.cc:2463] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:29.370558 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1a04c223889f424e831e82a52a84f804 (DEFAULT_TABLE table=default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.371007 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d616a8ad9ad4777b44deb77bc321d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:29.371259 847 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:29.372339 847 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:29.372442 847 leader_election.cc:304] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.372457 1164 raft_consensus.cc:357] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.372637 1164 raft_consensus.cc:383] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.372718 1164 raft_consensus.cc:738] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:29.372936 847 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 296703eedc6d459880a74b38e642bceb
I20250112 20:51:29.373116 1430 raft_consensus.cc:2743] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.373083 1164 consensus_queue.cc:260] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.373602 1164 ts_tablet_manager.cc:1428] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250112 20:51:29.373982 753 tablet_service.cc:1467] Processing CreateTablet for tablet c79ce3118cbf49e287c93a76398521da (DEFAULT_TABLE table=default.Table_1dd4ddaed71542489ede444900ece37b [id=c7f63522332840b6bc6cd7462015ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.371943 1186 ts_tablet_manager.cc:1918] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.374071 1164 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.374394 1186 log.cc:1198] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/382aefb4a5e34999b560a9237839b380
I20250112 20:51:29.374787 1186 ts_tablet_manager.cc:1939] T 382aefb4a5e34999b560a9237839b380 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.372051 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a04c223889f424e831e82a52a84f804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.375141 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 132) took 2448 ms. Trace:
I20250112 20:51:29.375260 1186 rpcz_store.cc:276] 0112 20:51:26.926838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.928410 (+ 1572us) service_pool.cc:225] Handling call
0112 20:51:29.087335 (+2158925us) ts_tablet_manager.cc:1145] Deleting tablet 382aefb4a5e34999b560a9237839b380
0112 20:51:29.087344 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.125816 (+ 38472us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.355994 (+230178us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.375103 (+ 19109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2158632}
I20250112 20:51:29.375676 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 133) took 2385 ms. Trace:
I20250112 20:51:29.375794 1186 rpcz_store.cc:276] 0112 20:51:26.990155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.990213 (+ 58us) service_pool.cc:225] Handling call
0112 20:51:29.375539 (+2385326us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:29.375549 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.375642 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2385118}
I20250112 20:51:29.376168 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 134) took 2380 ms. Trace:
I20250112 20:51:29.376283 1186 rpcz_store.cc:276] 0112 20:51:26.995808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.995861 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:29.376052 (+2380191us) ts_tablet_manager.cc:1145] Deleting tablet 1c7a54ce78854137ab2d12e9ce055565
0112 20:51:29.376060 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.376134 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2379995}
I20250112 20:51:29.376546 1186 tablet_replica.cc:331] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.376749 1186 raft_consensus.cc:2238] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.376979 1186 raft_consensus.cc:2267] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.377574 1186 ts_tablet_manager.cc:1905] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.383338 1186 ts_tablet_manager.cc:1918] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.383502 1186 log.cc:1198] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:29.383925 1186 ts_tablet_manager.cc:1939] T cc9aa4f42f89455cba7c72e7a49c6a94 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.384289 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 135) took 2385 ms. Trace:
I20250112 20:51:29.384433 1186 rpcz_store.cc:276] 0112 20:51:26.999139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:26.999196 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:29.376482 (+2377286us) ts_tablet_manager.cc:1145] Deleting tablet cc9aa4f42f89455cba7c72e7a49c6a94
0112 20:51:29.376494 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.380406 (+ 3912us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.383240 (+ 2834us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.384235 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377066}
I20250112 20:51:29.368996 1440 raft_consensus.cc:2743] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.370095 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296703eedc6d459880a74b38e642bceb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.384876 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 136) took 2361 ms. Trace:
I20250112 20:51:29.385025 1186 rpcz_store.cc:276] 0112 20:51:27.023166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.023604 (+ 438us) service_pool.cc:225] Handling call
0112 20:51:29.384679 (+2361075us) ts_tablet_manager.cc:1145] Deleting tablet 3ade9b410d8443249730cacdf7266eb2
0112 20:51:29.384687 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.384823 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2360835}
I20250112 20:51:29.385346 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 (DEFAULT_TABLE table=default.Table_b437c31235f14edb94ca1e295cb9487e [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.385408 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 137) took 2352 ms. Trace:
I20250112 20:51:29.385490 1186 rpcz_store.cc:276] 0112 20:51:27.033378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.033433 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:29.385279 (+2351846us) ts_tablet_manager.cc:1145] Deleting tablet 953d2ec8a7564c0bbcf0166487dd1746
0112 20:51:29.385287 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.385359 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2351676}
I20250112 20:51:29.385741 1186 tablet_replica.cc:331] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.385742 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b932b7e1e34d4ab18aae40047dc038. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.385108 1184 rpcz_store.cc:276] 0112 20:51:27.229799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.229964 (+ 165us) service_pool.cc:225] Handling call
0112 20:51:29.361131 (+2131167us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:29.368077 (+ 6946us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.368184 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130770}
I20250112 20:51:29.385906 1186 raft_consensus.cc:2238] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.386068 1186 raft_consensus.cc:2267] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.386462 750 tablet_service.cc:1467] Processing CreateTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 (DEFAULT_TABLE table=default.Table_b437c31235f14edb94ca1e295cb9487e [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:29.387419 847 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.387560 847 leader_election.cc:304] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.387835 1436 raft_consensus.cc:2743] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.388815 879 tablet_service.cc:1467] Processing CreateTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 (DEFAULT_TABLE table=default.Table_b437c31235f14edb94ca1e295cb9487e [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.372660 1163 tablet_bootstrap.cc:492] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.384624 1441 raft_consensus.cc:491] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.389169 1163 ts_tablet_manager.cc:1397] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.008s
I20250112 20:51:29.389250 1441 raft_consensus.cc:513] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.389477 1186 ts_tablet_manager.cc:1905] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.389725 1184 tablet_replica.cc:331] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:29.389915 1441 leader_election.cc:290] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.389941 1184 raft_consensus.cc:2238] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.390004 1163 raft_consensus.cc:357] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.390127 1184 raft_consensus.cc:2267] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.390175 1163 raft_consensus.cc:383] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.390262 1163 raft_consensus.cc:738] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.390352 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b932b7e1e34d4ab18aae40047dc038. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.390156 882 tablet_service.cc:1467] Processing CreateTablet for tablet c79ce3118cbf49e287c93a76398521da (DEFAULT_TABLE table=default.Table_1dd4ddaed71542489ede444900ece37b [id=c7f63522332840b6bc6cd7462015ad3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.390722 1184 ts_tablet_manager.cc:1905] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.390915 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79ce3118cbf49e287c93a76398521da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.391513 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea7336d13ea46748167e35e7e7e8c98" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:29.391728 913 raft_consensus.cc:2463] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:29.391842 1178 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.391984 1178 ts_tablet_manager.cc:1397] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.249s user 0.080s sys 0.092s
I20250112 20:51:29.392793 1178 raft_consensus.cc:357] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.393035 1178 raft_consensus.cc:383] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.393116 1178 raft_consensus.cc:738] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.393064 1163 consensus_queue.cc:260] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.393328 1178 consensus_queue.cc:260] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.393602 1163 ts_tablet_manager.cc:1428] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250112 20:51:29.393810 1178 ts_tablet_manager.cc:1428] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:29.394116 1178 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.394829 1163 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.395051 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.394837 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.395467 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f720bd9710d2476d9632699a2732cb05 (DEFAULT_TABLE table=default.Table_aba3e9f239704c2c843595403a218ca7 [id=32ea0a446f8e4aa2b87dfd1c5b691419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.395704 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.395882 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f720bd9710d2476d9632699a2732cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.397590 1178 tablet_bootstrap.cc:654] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.398273 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a04c223889f424e831e82a52a84f804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.398591 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b932b7e1e34d4ab18aae40047dc038. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.398779 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c79ce3118cbf49e287c93a76398521da. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:29.400390 983 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:29.400581 983 leader_election.cc:304] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.401273 1441 raft_consensus.cc:2798] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:29.401374 1441 raft_consensus.cc:491] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:29.401470 1441 raft_consensus.cc:3054] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.403347 887 tablet_service.cc:1467] Processing CreateTablet for tablet f720bd9710d2476d9632699a2732cb05 (DEFAULT_TABLE table=default.Table_aba3e9f239704c2c843595403a218ca7 [id=32ea0a446f8e4aa2b87dfd1c5b691419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.404029 1441 raft_consensus.cc:513] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.404513 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f (DEFAULT_TABLE table=default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.404543 752 tablet_service.cc:1467] Processing CreateTablet for tablet f720bd9710d2476d9632699a2732cb05 (DEFAULT_TABLE table=default.Table_aba3e9f239704c2c843595403a218ca7 [id=32ea0a446f8e4aa2b87dfd1c5b691419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.405124 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f (DEFAULT_TABLE table=default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.405406 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4301a07fcb044fd2a6f3e5ef424b411f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.406261 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f720bd9710d2476d9632699a2732cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.406738 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f (DEFAULT_TABLE table=default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.407537 1163 tablet_bootstrap.cc:654] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.407603 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b310e4135c2b424fb006a38a60e41066 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.408010 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b310e4135c2b424fb006a38a60e41066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.408994 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4301a07fcb044fd2a6f3e5ef424b411f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.416118 756 tablet_service.cc:1467] Processing CreateTablet for tablet b310e4135c2b424fb006a38a60e41066 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.416173 881 tablet_service.cc:1467] Processing CreateTablet for tablet b310e4135c2b424fb006a38a60e41066 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.416353 1048 raft_consensus.cc:3054] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.416524 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b310e4135c2b424fb006a38a60e41066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.416522 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b310e4135c2b424fb006a38a60e41066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.423799 1436 consensus_queue.cc:1035] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:29.425271 1164 tablet_bootstrap.cc:654] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.425585 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea7336d13ea46748167e35e7e7e8c98" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:29.425782 913 raft_consensus.cc:3054] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.426100 1022 tablet_service.cc:1467] Processing CreateTablet for tablet e912d588104f482498c40239c9e585b9 (DEFAULT_TABLE table=default.Table_74308835319d415ba7a13ff9e0a42d76 [id=c50492774af945b58a6ff2ba4b40db82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.426540 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e912d588104f482498c40239c9e585b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.428561 913 raft_consensus.cc:2463] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
W20250112 20:51:29.431298 983 leader_election.cc:343] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:29.431591 983 leader_election.cc:304] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.431934 1440 raft_consensus.cc:2798] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:29.432114 1440 raft_consensus.cc:695] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:29.432442 1440 consensus_queue.cc:237] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.433837 751 tablet_service.cc:1467] Processing CreateTablet for tablet e912d588104f482498c40239c9e585b9 (DEFAULT_TABLE table=default.Table_74308835319d415ba7a13ff9e0a42d76 [id=c50492774af945b58a6ff2ba4b40db82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.434248 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e912d588104f482498c40239c9e585b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.434996 881 tablet_service.cc:1467] Processing CreateTablet for tablet e912d588104f482498c40239c9e585b9 (DEFAULT_TABLE table=default.Table_74308835319d415ba7a13ff9e0a42d76 [id=c50492774af945b58a6ff2ba4b40db82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.435437 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e912d588104f482498c40239c9e585b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.441429 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (DEFAULT_TABLE table=default.Table_395d5be237c54baea392437bef2cbadf [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.441965 756 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (DEFAULT_TABLE table=default.Table_395d5be237c54baea392437bef2cbadf [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.442525 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e1c32cab2c4b438d3ff4a7e674feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.443276 876 tablet_service.cc:1514] Processing DeleteTablet for tablet fc63526f932841509277f4726303c801 with delete_type TABLET_DATA_DELETED (replaced by tablet d0829e9dd5854961b5402b5710624612) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.445678 882 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (DEFAULT_TABLE table=default.Table_395d5be237c54baea392437bef2cbadf [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.446640 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e1c32cab2c4b438d3ff4a7e674feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.447388 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e1c32cab2c4b438d3ff4a7e674feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.448838 1441 leader_election.cc:290] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:29.453606 1427 raft_consensus.cc:491] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.453773 1427 raft_consensus.cc:513] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.454392 1427 leader_election.cc:290] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.455549 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:29.455567 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.456128 710 leader_election.cc:343] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:29.456346 710 leader_election.cc:343] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.456432 710 leader_election.cc:304] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.456691 1427 raft_consensus.cc:2743] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.457871 1163 tablet_bootstrap.cc:492] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.457944 1185 ts_tablet_manager.cc:1918] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.458060 1185 log.cc:1198] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4d616a8ad9ad4777b44deb77bc321d66
I20250112 20:51:29.458102 1163 ts_tablet_manager.cc:1397] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.006s
I20250112 20:51:29.458505 1185 ts_tablet_manager.cc:1939] T 4d616a8ad9ad4777b44deb77bc321d66 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.458998 1163 raft_consensus.cc:357] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.459218 1163 raft_consensus.cc:383] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.459293 1163 raft_consensus.cc:738] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.459543 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f720bd9710d2476d9632699a2732cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.459540 1163 consensus_queue.cc:260] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.460089 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4301a07fcb044fd2a6f3e5ef424b411f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.460089 1163 ts_tablet_manager.cc:1428] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:29.460952 1163 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.462572 1163 tablet_bootstrap.cc:654] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.489581 1164 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.492058 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet fc63526f932841509277f4726303c801 with delete_type TABLET_DATA_DELETED (replaced by tablet d0829e9dd5854961b5402b5710624612) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:29.495388 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3ade9b410d8443249730cacdf7266eb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ade9b410d8443249730cacdf7266eb2 already in progress: creating tablet
I20250112 20:51:29.495806 1164 ts_tablet_manager.cc:1397] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.000s
I20250112 20:51:29.500110 1164 raft_consensus.cc:357] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.500990 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
W20250112 20:51:29.503700 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ed45eb5cbd2e489c83c3d343ca1adcfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed45eb5cbd2e489c83c3d343ca1adcfc already in progress: creating tablet
W20250112 20:51:29.507382 847 consensus_peers.cc:487] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7a8468369af418ea4a1b0a681f8ff74. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:29.512451 1164 raft_consensus.cc:383] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.491011 753 tablet_service.cc:1514] Processing DeleteTablet for tablet fc63526f932841509277f4726303c801 with delete_type TABLET_DATA_DELETED (replaced by tablet d0829e9dd5854961b5402b5710624612) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.513005 1185 tablet_replica.cc:331] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:29.513294 1185 raft_consensus.cc:2238] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.513844 1185 pending_rounds.cc:62] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:29.514010 1185 pending_rounds.cc:69] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113585006422933504 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "fc63526f932841509277f4726303c801" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" new_extra_config { } }
W20250112 20:51:29.514598 1185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47160: Aborted: Op aborted
I20250112 20:51:29.515326 1185 raft_consensus.cc:2267] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:29.516515 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 296703eedc6d459880a74b38e642bceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 296703eedc6d459880a74b38e642bceb already in progress: creating tablet
I20250112 20:51:29.516749 1164 raft_consensus.cc:738] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.517866 1178 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.518038 1178 ts_tablet_manager.cc:1397] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.124s user 0.011s sys 0.025s
I20250112 20:51:29.518846 1178 raft_consensus.cc:357] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.517207 1164 consensus_queue.cc:260] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.520313 1178 raft_consensus.cc:383] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.520463 1178 raft_consensus.cc:738] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.520742 1178 consensus_queue.cc:260] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.521479 1178 ts_tablet_manager.cc:1428] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:29.522034 1178 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.523950 1178 tablet_bootstrap.cc:654] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.522099 1186 ts_tablet_manager.cc:1918] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.524381 1186 log.cc:1198] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:29.524852 1186 ts_tablet_manager.cc:1939] T ed45eb5cbd2e489c83c3d343ca1adcfc P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.525627 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 138) took 2488 ms. Trace:
I20250112 20:51:29.525784 1186 rpcz_store.cc:276] 0112 20:51:27.036706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.036759 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:29.385674 (+2348915us) ts_tablet_manager.cc:1145] Deleting tablet ed45eb5cbd2e489c83c3d343ca1adcfc
0112 20:51:29.385682 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.467118 (+ 81436us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.521625 (+ 54507us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.525595 (+ 3970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348691}
I20250112 20:51:29.526463 1186 tablet_replica.cc:331] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.526696 1186 raft_consensus.cc:2238] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.526895 1186 raft_consensus.cc:2267] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.527638 1164 ts_tablet_manager.cc:1428] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250112 20:51:29.528108 1186 ts_tablet_manager.cc:1905] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:29.528112 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b32dc02497f64a2e9ad99da6c9115d5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32dc02497f64a2e9ad99da6c9115d5d already in progress: creating tablet
I20250112 20:51:29.537073 1164 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.537662 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 313a5c9905bc445893f1ffa8752ca589 (table default.Table_6a23debfb16c4480b703dc234f76eae5 [id=86a3a89b9ffa4029af0b95ab7b0b6f70]) successfully deleted
I20250112 20:51:29.540297 1184 ts_tablet_manager.cc:1918] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250112 20:51:29.543445 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 00c0712ab2de43eea8c77ab39a971a2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c0712ab2de43eea8c77ab39a971a2c already in progress: creating tablet
I20250112 20:51:29.547438 1164 tablet_bootstrap.cc:654] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.549816 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_133503fd7b744d8f8e3db0843e4ac9e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.550392 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133503fd7b744d8f8e3db0843e4ac9e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.554580 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0afc27e58eec42daadf528a66340c624" } modify_external_catalogs: true
I20250112 20:51:29.554786 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0afc27e58eec42daadf528a66340c624" } modify_external_catalogs: true
I20250112 20:51:29.555612 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.555886 1184 log.cc:1198] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e040383f6e2a4ccf9614fd2d119a3cb4
I20250112 20:51:29.556166 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9e27bc50d3ea443ab9b522fe653d2269" } new_table_name: "default.Table_c62b92e5784e44af9e27e902aa48a6ca" modify_external_catalogs: true
I20250112 20:51:29.556318 1184 ts_tablet_manager.cc:1939] T e040383f6e2a4ccf9614fd2d119a3cb4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.557193 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 148) took 2240 ms. Trace:
I20250112 20:51:29.557286 1184 rpcz_store.cc:276] 0112 20:51:27.316553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.323342 (+ 6789us) service_pool.cc:225] Handling call
0112 20:51:29.389649 (+2066307us) ts_tablet_manager.cc:1145] Deleting tablet e040383f6e2a4ccf9614fd2d119a3cb4
0112 20:51:29.389659 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.497613 (+107954us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.540172 (+ 42559us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.557137 (+ 16965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631070}
I20250112 20:51:29.557417 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8df177b5b9c84139b3ac6874e7c7fd1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.557890 1186 ts_tablet_manager.cc:1918] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:29.557891 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df177b5b9c84139b3ac6874e7c7fd1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.558029 1186 log.cc:1198] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/296703eedc6d459880a74b38e642bceb
I20250112 20:51:29.558310 1186 ts_tablet_manager.cc:1939] T 296703eedc6d459880a74b38e642bceb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.558992 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 139) took 2518 ms. Trace:
I20250112 20:51:29.559079 1186 rpcz_store.cc:276] 0112 20:51:27.040395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.040444 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:29.526410 (+2485966us) ts_tablet_manager.cc:1145] Deleting tablet 296703eedc6d459880a74b38e642bceb
0112 20:51:29.526418 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.553964 (+ 27546us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.557791 (+ 3827us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.558932 (+ 1141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2485804}
I20250112 20:51:29.559098 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_53ad9b3fe99c4e67ad2569718afe2e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.559562 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53ad9b3fe99c4e67ad2569718afe2e16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.542784 1185 ts_tablet_manager.cc:1905] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.560672 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c3afc9f0dcf0422fa7cdd946feec0e58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:29.561180 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3afc9f0dcf0422fa7cdd946feec0e58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.562141 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.564721 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_18e26eeea5ff4c4196d49055411fb3ad" } modify_external_catalogs: true
I20250112 20:51:29.564896 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_18e26eeea5ff4c4196d49055411fb3ad" } modify_external_catalogs: true
I20250112 20:51:29.566061 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_aba3e9f239704c2c843595403a218ca7" } new_table_name: "default.Table_7fdddeb1e3534f599b5045b3a517b64c" modify_external_catalogs: true
I20250112 20:51:29.566162 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.566169 1178 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.566387 1178 ts_tablet_manager.cc:1397] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.004s
I20250112 20:51:29.566777 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.567417 1178 raft_consensus.cc:357] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.567646 1178 raft_consensus.cc:383] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.567770 1178 raft_consensus.cc:738] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.567854 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_69153618434846ad9d90837893bea2da" } modify_external_catalogs: true
I20250112 20:51:29.568030 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_69153618434846ad9d90837893bea2da" } modify_external_catalogs: true
I20250112 20:51:29.568143 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.568066 1178 consensus_queue.cc:260] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.568827 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.569478 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.570420 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.570816 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 163) took 2207 ms. Trace:
I20250112 20:51:29.571488 1178 ts_tablet_manager.cc:1428] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:29.571777 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.575591 1178 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.576303 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd46a422093495ba8b5b03039874a32 (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.576689 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd46a422093495ba8b5b03039874a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.577242 1178 tablet_bootstrap.cc:654] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.581586 1186 tablet_replica.cc:331] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.582001 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 64cd83b199bb4da28d39da449945c389
I20250112 20:51:29.585155 1186 raft_consensus.cc:2238] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.585579 1186 pending_rounds.cc:62] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:29.585659 1186 pending_rounds.cc:69] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113584999901016064 op_type: NO_OP noop_request { }
I20250112 20:51:29.585822 1186 raft_consensus.cc:2883] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:29.585940 1186 raft_consensus.cc:2267] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.574836 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.580807 1048 raft_consensus.cc:3054] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.586635 1186 ts_tablet_manager.cc:1905] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.587440 1178 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.587570 1178 ts_tablet_manager.cc:1397] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20250112 20:51:29.588388 1178 raft_consensus.cc:357] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.588593 1178 raft_consensus.cc:383] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.588672 1178 raft_consensus.cc:738] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.588920 1178 consensus_queue.cc:260] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.592375 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ef68d20727c04b09980aac02d08a81bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.593410 1163 tablet_bootstrap.cc:492] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.593550 1163 ts_tablet_manager.cc:1397] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.133s user 0.041s sys 0.036s
I20250112 20:51:29.593571 1178 ts_tablet_manager.cc:1428] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:29.594328 1163 raft_consensus.cc:357] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.594558 1163 raft_consensus.cc:383] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.594655 1163 raft_consensus.cc:738] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.595702 1184 rpcz_store.cc:276] 0112 20:51:27.363840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.363920 (+ 80us) service_pool.cc:225] Handling call
0112 20:51:29.570627 (+2206707us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:29.570640 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.570764 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1811778}
I20250112 20:51:29.596343 1178 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.596392 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_21b7926d70e24852a7e66dbb6449e4ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:29.596707 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd46a422093495ba8b5b03039874a32 (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:29.597005 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21b7926d70e24852a7e66dbb6449e4ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.597273 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd46a422093495ba8b5b03039874a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.599251 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (DEFAULT_TABLE table=default.Table_d184983b685e4a4cb571a72c242ef9b3 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.599669 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.602226 887 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (DEFAULT_TABLE table=default.Table_d184983b685e4a4cb571a72c242ef9b3 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.602605 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:29.602600 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef68d20727c04b09980aac02d08a81bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:29.604561 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (DEFAULT_TABLE table=default.Table_d184983b685e4a4cb571a72c242ef9b3 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.604686 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd46a422093495ba8b5b03039874a32 (DEFAULT_TABLE table=default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.605046 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.605204 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd46a422093495ba8b5b03039874a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.605365 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5d384628ba9b4f81b768eda4f796632b (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.605703 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d384628ba9b4f81b768eda4f796632b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:29.609077 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83c13126e53b46c0a1e7beafe1b0d295 consensus state has no leader
I20250112 20:51:29.613603 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b4ad7cf1e0c74265a4d21931279f2318 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:29.615906 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5d384628ba9b4f81b768eda4f796632b (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.616286 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d384628ba9b4f81b768eda4f796632b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.616356 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf1dc64f1964202bfa8169388a34781 (DEFAULT_TABLE table=default.Table_8399a4497e68459fb8e495749b55f0a3 [id=7e2c24519e17479fb708264c8543222b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.616652 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf1dc64f1964202bfa8169388a34781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.618578 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 164) took 2254 ms. Trace:
I20250112 20:51:29.618742 1184 rpcz_store.cc:276] 0112 20:51:27.364069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.364132 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:29.598501 (+2234369us) ts_tablet_manager.cc:1145] Deleting tablet af1aac31fd32400b9094f8e2de4fe42c
0112 20:51:29.598510 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.616692 (+ 18182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815284}
I20250112 20:51:29.619441 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 165) took 2255 ms. Trace:
I20250112 20:51:29.619709 1184 rpcz_store.cc:276] 0112 20:51:27.364227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.368559 (+ 4332us) service_pool.cc:225] Handling call
0112 20:51:29.619292 (+2250733us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:29.619304 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.619388 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1835207}
I20250112 20:51:29.620227 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 176) took 2122 ms. Trace:
I20250112 20:51:29.620394 1184 rpcz_store.cc:276] 0112 20:51:27.497722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.928346 (+430624us) service_pool.cc:225] Handling call
0112 20:51:29.620076 (+1691730us) ts_tablet_manager.cc:1145] Deleting tablet 6fe28f23448c47d3979f1982f939e39c
0112 20:51:29.620089 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.620171 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690402}
I20250112 20:51:29.620975 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 177) took 2023 ms. Trace:
I20250112 20:51:29.621567 1184 rpcz_store.cc:276] 0112 20:51:27.597694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.929700 (+332006us) service_pool.cc:225] Handling call
0112 20:51:29.620822 (+1691122us) ts_tablet_manager.cc:1145] Deleting tablet 0e2235e2a63c4e1192b3260085908445
0112 20:51:29.620837 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.620914 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690929}
I20250112 20:51:29.622076 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 178) took 1948 ms. Trace:
I20250112 20:51:29.622958 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5d384628ba9b4f81b768eda4f796632b (DEFAULT_TABLE table=default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.623450 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d384628ba9b4f81b768eda4f796632b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.623826 1184 rpcz_store.cc:276] 0112 20:51:27.673296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.929860 (+256564us) service_pool.cc:225] Handling call
0112 20:51:29.621934 (+1692074us) ts_tablet_manager.cc:1145] Deleting tablet b84624dc685b401192ca753ff59da016
0112 20:51:29.621942 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.622024 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691540}
I20250112 20:51:29.624769 1436 raft_consensus.cc:491] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.625164 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 179) took 1949 ms. Trace:
I20250112 20:51:29.625303 1184 rpcz_store.cc:276] 0112 20:51:27.675875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.930414 (+254539us) service_pool.cc:225] Handling call
0112 20:51:29.625000 (+1694586us) ts_tablet_manager.cc:1145] Deleting tablet 8b414be57990432bb786daf5faf0cc2e
0112 20:51:29.625009 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.625120 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694427}
I20250112 20:51:29.626456 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 180) took 1949 ms. Trace:
I20250112 20:51:29.626590 1184 rpcz_store.cc:276] 0112 20:51:27.677190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.930590 (+253400us) service_pool.cc:225] Handling call
0112 20:51:29.625825 (+1695235us) ts_tablet_manager.cc:1145] Deleting tablet d8330fa1e4654a1a9f167a1120a103fc
0112 20:51:29.625833 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.626397 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687986}
I20250112 20:51:29.626849 1436 raft_consensus.cc:513] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.627439 1427 raft_consensus.cc:491] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.627666 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 191) took 1204 ms. Trace:
I20250112 20:51:29.627624 1427 raft_consensus.cc:513] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.627822 1184 rpcz_store.cc:276] 0112 20:51:28.423470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.423522 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:29.627469 (+1203947us) ts_tablet_manager.cc:1145] Deleting tablet cc971026f6b44db6af8215890ba71075
0112 20:51:29.627478 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.627601 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203695}
I20250112 20:51:29.628401 1427 leader_election.cc:290] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.628441 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 194) took 1041 ms. Trace:
I20250112 20:51:29.628886 1178 tablet_bootstrap.cc:654] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.628989 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.629868 710 leader_election.cc:343] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.628631 1184 rpcz_store.cc:276] 0112 20:51:28.586652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.587248 (+ 596us) service_pool.cc:225] Handling call
0112 20:51:29.628311 (+1041063us) ts_tablet_manager.cc:1145] Deleting tablet 0871d4272e874a958174f96bfc0e1a95
0112 20:51:29.628319 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.628397 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040804}
I20250112 20:51:29.630520 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 195) took 1018 ms. Trace:
I20250112 20:51:29.630646 1184 rpcz_store.cc:276] 0112 20:51:28.612377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.615518 (+ 3141us) service_pool.cc:225] Handling call
0112 20:51:29.630398 (+1014880us) ts_tablet_manager.cc:1145] Deleting tablet 6f4421d16e3f423794c44970a2469b10
0112 20:51:29.630406 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.630474 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014641}
I20250112 20:51:29.631222 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 196) took 1018 ms. Trace:
I20250112 20:51:29.631470 1184 rpcz_store.cc:276] 0112 20:51:28.612502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.615797 (+ 3295us) service_pool.cc:225] Handling call
0112 20:51:29.630985 (+1015188us) ts_tablet_manager.cc:1145] Deleting tablet 201a7e2f17bb46fa918dad8f878a4065
0112 20:51:29.630993 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.631175 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015006}
I20250112 20:51:29.631976 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 197) took 1012 ms. Trace:
I20250112 20:51:29.631917 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf1dc64f1964202bfa8169388a34781 (DEFAULT_TABLE table=default.Table_8399a4497e68459fb8e495749b55f0a3 [id=7e2c24519e17479fb708264c8543222b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.632114 1184 rpcz_store.cc:276] 0112 20:51:28.619029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.619872 (+ 843us) service_pool.cc:225] Handling call
0112 20:51:29.631844 (+1011972us) ts_tablet_manager.cc:1145] Deleting tablet e0f1c76db0d64624bd43f0e538f0649d
0112 20:51:29.631854 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.631930 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010085}
I20250112 20:51:29.632314 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf1dc64f1964202bfa8169388a34781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.632617 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 198) took 1012 ms. Trace:
I20250112 20:51:29.632766 1184 rpcz_store.cc:276] 0112 20:51:28.620306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.622355 (+ 2049us) service_pool.cc:225] Handling call
0112 20:51:29.632481 (+1010126us) ts_tablet_manager.cc:1145] Deleting tablet f249d70b116443b7823e093efeda5cc3
0112 20:51:29.632488 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.632572 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009943}
I20250112 20:51:29.634061 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:29.634290 775 raft_consensus.cc:2463] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:29.635350 1184 tablet_replica.cc:331] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:29.635854 1184 raft_consensus.cc:2238] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.637395 847 leader_election.cc:304] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:29.638021 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.638135 1436 leader_election.cc:290] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.638410 1430 raft_consensus.cc:2798] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:29.638520 1430 raft_consensus.cc:491] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:29.638520 1184 raft_consensus.cc:2267] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.638638 1430 raft_consensus.cc:3054] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.639051 1184 ts_tablet_manager.cc:1905] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.637737 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.641419 710 leader_election.cc:343] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:29.641494 847 leader_election.cc:343] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.641556 710 leader_election.cc:304] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.641820 1427 raft_consensus.cc:2743] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.650975 1186 ts_tablet_manager.cc:1918] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:29.651279 1186 log.cc:1198] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d48b17628cf145b69e5fdde4131d9fd2
I20250112 20:51:29.651894 1186 ts_tablet_manager.cc:1939] T d48b17628cf145b69e5fdde4131d9fd2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.652343 1430 raft_consensus.cc:513] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.652706 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 140) took 2582 ms. Trace:
I20250112 20:51:29.653367 1186 rpcz_store.cc:276] 0112 20:51:27.069990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.070047 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:29.581477 (+2511430us) ts_tablet_manager.cc:1145] Deleting tablet d48b17628cf145b69e5fdde4131d9fd2
0112 20:51:29.581491 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.602785 (+ 21294us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.633373 (+ 30588us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.652655 (+ 19282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2511187}
I20250112 20:51:29.653759 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 141) took 2552 ms. Trace:
I20250112 20:51:29.654014 1430 leader_election.cc:290] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.654645 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:29.654729 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4390c60d6c724190a216ef9c961b9592" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:29.654955 775 raft_consensus.cc:3054] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.655341 1186 rpcz_store.cc:276] 0112 20:51:27.101132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.101194 (+ 62us) service_pool.cc:225] Handling call
0112 20:51:29.653604 (+2552410us) ts_tablet_manager.cc:1145] Deleting tablet b32dc02497f64a2e9ad99da6c9115d5d
0112 20:51:29.653612 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.653700 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2552200}
I20250112 20:51:29.656042 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 149) took 2424 ms. Trace:
I20250112 20:51:29.656504 1186 rpcz_store.cc:276] 0112 20:51:27.231610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.231656 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:29.655889 (+2424233us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:29.655898 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.655999 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424020}
I20250112 20:51:29.657048 1186 tablet_replica.cc:331] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.657483 1186 raft_consensus.cc:2238] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.657610 775 raft_consensus.cc:2463] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:29.657964 1186 raft_consensus.cc:2267] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.658764 1186 ts_tablet_manager.cc:1905] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.660841 1184 ts_tablet_manager.cc:1918] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.661219 847 leader_election.cc:304] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:29.661582 847 leader_election.cc:343] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.662117 1436 raft_consensus.cc:2798] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:29.666952 1184 log.cc:1198] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/719b906973a8490898d6befce49910d3
I20250112 20:51:29.667006 1186 ts_tablet_manager.cc:1918] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:29.667476 1184 ts_tablet_manager.cc:1939] T 719b906973a8490898d6befce49910d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.669612 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02
I20250112 20:51:29.670114 450 catalog_manager.cc:6596] ReplaceTablet: tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 deleted and replaced by tablet d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:29.670552 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2b8250621cbe45eba669b5ef14e672c2
I20250112 20:51:29.671888 1186 log.cc:1198] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e040383f6e2a4ccf9614fd2d119a3cb4
I20250112 20:51:29.672509 1186 ts_tablet_manager.cc:1939] T e040383f6e2a4ccf9614fd2d119a3cb4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.673813 1184 tablet_replica.cc:331] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:29.673847 1430 raft_consensus.cc:491] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.674116 1184 raft_consensus.cc:2238] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.674100 1430 raft_consensus.cc:513] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.674422 1184 pending_rounds.cc:62] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:29.674306 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 150) took 2396 ms. Trace:
I20250112 20:51:29.674542 1184 pending_rounds.cc:69] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585004846252032 op_type: NO_OP noop_request { }
I20250112 20:51:29.674672 1184 raft_consensus.cc:2883] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:29.674814 1184 raft_consensus.cc:2267] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.674863 1430 leader_election.cc:290] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.674624 1186 rpcz_store.cc:276] 0112 20:51:27.278301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.278382 (+ 81us) service_pool.cc:225] Handling call
0112 20:51:29.656960 (+2378578us) ts_tablet_manager.cc:1145] Deleting tablet e040383f6e2a4ccf9614fd2d119a3cb4
0112 20:51:29.656969 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.663989 (+ 7020us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.666887 (+ 2898us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.674265 (+ 7378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2378368}
I20250112 20:51:29.675766 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 165) took 2233 ms. Trace:
I20250112 20:51:29.675830 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d616a8ad9ad4777b44deb77bc321d66" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.676270 1186 rpcz_store.cc:276] 0112 20:51:27.442041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.442534 (+ 493us) service_pool.cc:225] Handling call
0112 20:51:29.675639 (+2233105us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:29.675648 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.675736 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232787}
I20250112 20:51:29.676717 1184 ts_tablet_manager.cc:1905] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.677121 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 167) took 2234 ms. Trace:
I20250112 20:51:29.677495 1186 rpcz_store.cc:276] 0112 20:51:27.442263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.444863 (+ 2600us) service_pool.cc:225] Handling call
0112 20:51:29.676979 (+2232116us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:29.676989 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.677092 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231822}
I20250112 20:51:29.678095 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 166) took 2235 ms. Trace:
W20250112 20:51:29.676110 847 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4d616a8ad9ad4777b44deb77bc321d66
I20250112 20:51:29.678521 1186 rpcz_store.cc:276] 0112 20:51:27.442163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.446052 (+ 3889us) service_pool.cc:225] Handling call
0112 20:51:29.677973 (+2231921us) ts_tablet_manager.cc:1145] Deleting tablet af1aac31fd32400b9094f8e2de4fe42c
0112 20:51:29.677981 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.678067 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231604}
I20250112 20:51:29.679095 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 178) took 2081 ms. Trace:
I20250112 20:51:29.679483 1186 rpcz_store.cc:276] 0112 20:51:27.597654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.597750 (+ 96us) service_pool.cc:225] Handling call
0112 20:51:29.678983 (+2081233us) ts_tablet_manager.cc:1145] Deleting tablet 6fe28f23448c47d3979f1982f939e39c
0112 20:51:29.678991 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.679061 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059527}
W20250112 20:51:29.679586 847 leader_election.cc:343] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.679939 847 leader_election.cc:304] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.680231 1430 raft_consensus.cc:2743] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.680687 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 180) took 1817 ms. Trace:
I20250112 20:51:29.681098 1186 rpcz_store.cc:276] 0112 20:51:27.862743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.866822 (+ 4079us) service_pool.cc:225] Handling call
0112 20:51:29.680554 (+1813732us) ts_tablet_manager.cc:1145] Deleting tablet b84624dc685b401192ca753ff59da016
0112 20:51:29.680564 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.680661 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813486}
I20250112 20:51:29.681864 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 181) took 1819 ms. Trace:
I20250112 20:51:29.682308 1186 rpcz_store.cc:276] 0112 20:51:27.862875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.867106 (+ 4231us) service_pool.cc:225] Handling call
0112 20:51:29.681748 (+1814642us) ts_tablet_manager.cc:1145] Deleting tablet 8b414be57990432bb786daf5faf0cc2e
0112 20:51:29.681757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.681829 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814505}
I20250112 20:51:29.683012 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 182) took 1820 ms. Trace:
I20250112 20:51:29.683419 1186 rpcz_store.cc:276] 0112 20:51:27.862993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:27.867264 (+ 4271us) service_pool.cc:225] Handling call
0112 20:51:29.682870 (+1815606us) ts_tablet_manager.cc:1145] Deleting tablet d8330fa1e4654a1a9f167a1120a103fc
0112 20:51:29.682897 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.682976 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815455}
W20250112 20:51:29.683501 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f720bd9710d2476d9632699a2732cb05 consensus state has no leader
I20250112 20:51:29.684106 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 179) took 1821 ms. Trace:
I20250112 20:51:29.684531 1186 rpcz_store.cc:276] 0112 20:51:27.862620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.001734 (+139114us) service_pool.cc:225] Handling call
0112 20:51:29.683995 (+1682261us) ts_tablet_manager.cc:1145] Deleting tablet 0e2235e2a63c4e1192b3260085908445
0112 20:51:29.684004 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.684079 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274883}
I20250112 20:51:29.685204 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 193) took 1243 ms. Trace:
I20250112 20:51:29.685608 1186 rpcz_store.cc:276] 0112 20:51:28.441717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.444837 (+ 3120us) service_pool.cc:225] Handling call
0112 20:51:29.685096 (+1240259us) ts_tablet_manager.cc:1145] Deleting tablet cc971026f6b44db6af8215890ba71075
0112 20:51:29.685104 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.685178 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240098}
I20250112 20:51:29.686169 1436 raft_consensus.cc:695] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:29.686218 1186 tablet_replica.cc:331] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.686652 1186 raft_consensus.cc:2238] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.686524 1436 consensus_queue.cc:237] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.688474 1186 raft_consensus.cc:2267] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.690456 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet bcbbefa7b55041a6a6b50b3c96c45833
I20250112 20:51:29.695024 1184 ts_tablet_manager.cc:1918] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:29.696522 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b4ad7cf1e0c74265a4d21931279f2318
I20250112 20:51:29.696944 453 catalog_manager.cc:6596] ReplaceTablet: tablet b4ad7cf1e0c74265a4d21931279f2318 deleted and replaced by tablet 35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:29.699664 1427 raft_consensus.cc:491] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.699824 1427 raft_consensus.cc:513] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.700479 1427 leader_election.cc:290] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.703029 1184 log.cc:1198] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4d616a8ad9ad4777b44deb77bc321d66
I20250112 20:51:29.701707 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.703783 1184 ts_tablet_manager.cc:1939] T 4d616a8ad9ad4777b44deb77bc321d66 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.706164 1184 tablet_replica.cc:331] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:29.706143 710 leader_election.cc:343] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.706885 1186 ts_tablet_manager.cc:1905] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.710074 887 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf1dc64f1964202bfa8169388a34781 (DEFAULT_TABLE table=default.Table_8399a4497e68459fb8e495749b55f0a3 [id=7e2c24519e17479fb708264c8543222b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.709224 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.712284 710 leader_election.cc:343] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.712459 710 leader_election.cc:304] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.713105 1427 raft_consensus.cc:2743] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.716190 751 tablet_service.cc:1467] Processing CreateTablet for tablet f3facda2ee004b2e9487194ba040c9cf (DEFAULT_TABLE table=default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0 [id=41df221c4bfb42fe8cd85808401151b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.716578 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3facda2ee004b2e9487194ba040c9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.716715 890 tablet_service.cc:1467] Processing CreateTablet for tablet f3facda2ee004b2e9487194ba040c9cf (DEFAULT_TABLE table=default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0 [id=41df221c4bfb42fe8cd85808401151b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.717430 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf1dc64f1964202bfa8169388a34781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.718083 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3facda2ee004b2e9487194ba040c9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.724996 1184 raft_consensus.cc:2238] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.717799 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d0829e9dd5854961b5402b5710624612 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.725379 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0829e9dd5854961b5402b5710624612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.727540 1186 ts_tablet_manager.cc:1918] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.727689 1186 log.cc:1198] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/0871d4272e874a958174f96bfc0e1a95
I20250112 20:51:29.728056 1186 ts_tablet_manager.cc:1939] T 0871d4272e874a958174f96bfc0e1a95 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.715319 1018 tablet_service.cc:1467] Processing CreateTablet for tablet f3facda2ee004b2e9487194ba040c9cf (DEFAULT_TABLE table=default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0 [id=41df221c4bfb42fe8cd85808401151b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.733135 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 196) took 1146 ms. Trace:
I20250112 20:51:29.733162 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3facda2ee004b2e9487194ba040c9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.733247 1186 rpcz_store.cc:276] 0112 20:51:28.586901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.586965 (+ 64us) service_pool.cc:225] Handling call
0112 20:51:29.686158 (+1099193us) ts_tablet_manager.cc:1145] Deleting tablet 0871d4272e874a958174f96bfc0e1a95
0112 20:51:29.686166 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.724535 (+ 38369us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.727440 (+ 2905us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.733077 (+ 5637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098964}
I20250112 20:51:29.733893 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 197) took 1120 ms. Trace:
I20250112 20:51:29.734055 1186 rpcz_store.cc:276] 0112 20:51:28.613012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.613065 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:29.733729 (+1120664us) ts_tablet_manager.cc:1145] Deleting tablet 6f4421d16e3f423794c44970a2469b10
0112 20:51:29.733738 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.733837 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120442}
I20250112 20:51:29.734615 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 198) took 1121 ms. Trace:
I20250112 20:51:29.734762 1186 rpcz_store.cc:276] 0112 20:51:28.613146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.613323 (+ 177us) service_pool.cc:225] Handling call
0112 20:51:29.734460 (+1121137us) ts_tablet_manager.cc:1145] Deleting tablet 201a7e2f17bb46fa918dad8f878a4065
0112 20:51:29.734469 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.734562 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121002}
I20250112 20:51:29.735221 1186 tablet_replica.cc:331] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.735482 1186 raft_consensus.cc:2238] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.735720 1186 raft_consensus.cc:2267] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.736542 1186 ts_tablet_manager.cc:1905] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.738479 1185 ts_tablet_manager.cc:1918] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:29.738831 1163 consensus_queue.cc:260] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.739358 1184 raft_consensus.cc:2267] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.739557 1163 ts_tablet_manager.cc:1428] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.146s user 0.001s sys 0.001s
I20250112 20:51:29.739876 1163 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.740676 1184 ts_tablet_manager.cc:1905] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.744370 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 64887b3578284cb286ee122ab8c626c4 (DEFAULT_TABLE table=default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.744814 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64887b3578284cb286ee122ab8c626c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.742758 1186 ts_tablet_manager.cc:1918] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.747953 1186 log.cc:1198] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:29.748344 1186 ts_tablet_manager.cc:1939] T e0f1c76db0d64624bd43f0e538f0649d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:29.743605 887 tablet_service.cc:1467] Processing CreateTablet for tablet d0829e9dd5854961b5402b5710624612 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.749141 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 199) took 1129 ms. Trace:
I20250112 20:51:29.749209 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0829e9dd5854961b5402b5710624612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.751533 1184 ts_tablet_manager.cc:1918] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:29.757125 1184 log.cc:1198] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/fc63526f932841509277f4726303c801
I20250112 20:51:29.757628 1184 ts_tablet_manager.cc:1939] T fc63526f932841509277f4726303c801 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:29.765666 890 tablet_service.cc:1467] Processing CreateTablet for tablet 64887b3578284cb286ee122ab8c626c4 (DEFAULT_TABLE table=default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.766142 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64887b3578284cb286ee122ab8c626c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.769927 453 catalog_manager.cc:5526] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:29.770360 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 4390c60d6c724190a216ef9c961b9592 (table default.Table_9c771feed27441ec8340f7279a4a06c1 [id=2ee1f97fef0f4e2b83932132d0936504]). Expected version 2 got 0
I20250112 20:51:29.777206 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 (DEFAULT_TABLE table=default.Table_90937bd663d44575baa256f844890ba9 [id=942d944d07b146fbb1e30e593fa0a266]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.777653 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6d5d425e6a493daa46900cbd8db166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.777915 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 (DEFAULT_TABLE table=default.Table_90937bd663d44575baa256f844890ba9 [id=942d944d07b146fbb1e30e593fa0a266]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.778283 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6d5d425e6a493daa46900cbd8db166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.785302 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 (DEFAULT_TABLE table=default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.785761 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944b71897bcf4304bd9f7bb7adef34e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.785844 881 tablet_service.cc:1467] Processing CreateTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 (DEFAULT_TABLE table=default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.786204 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944b71897bcf4304bd9f7bb7adef34e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.787318 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 382aefb4a5e34999b560a9237839b380 (table default.Table_f8ae4b86db714d6bba40875699ee81a8 [id=dc615ace8dab471b9fd1fa2ba79309d8]) successfully deleted
W20250112 20:51:29.787569 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 304e1c8abe094bd99a99fd358b697f81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 304e1c8abe094bd99a99fd358b697f81 already in progress: creating tablet
W20250112 20:51:29.787798 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1c7a54ce78854137ab2d12e9ce055565 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c7a54ce78854137ab2d12e9ce055565 already in progress: creating tablet
I20250112 20:51:29.787990 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet cc9aa4f42f89455cba7c72e7a49c6a94 (table default.Table_9cf0195d130e459fbc8e651f57546be7 [id=7bde50a83284433daf462fe09db9061c]) successfully deleted
W20250112 20:51:29.788175 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3ade9b410d8443249730cacdf7266eb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ade9b410d8443249730cacdf7266eb2 already in progress: creating tablet
I20250112 20:51:29.742920 756 tablet_service.cc:1467] Processing CreateTablet for tablet d0829e9dd5854961b5402b5710624612 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.788697 749 tablet_service.cc:1467] Processing CreateTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 (DEFAULT_TABLE table=default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.788957 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0829e9dd5854961b5402b5710624612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.789137 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944b71897bcf4304bd9f7bb7adef34e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.790266 1164 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.790793 1164 ts_tablet_manager.cc:1397] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.254s user 0.002s sys 0.005s
I20250112 20:51:29.791621 1164 raft_consensus.cc:357] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.791841 1164 raft_consensus.cc:383] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.791922 1164 raft_consensus.cc:738] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.792132 1164 consensus_queue.cc:260] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.792659 1178 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.766629 751 tablet_service.cc:1467] Processing CreateTablet for tablet 64887b3578284cb286ee122ab8c626c4 (DEFAULT_TABLE table=default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.772912 1427 raft_consensus.cc:491] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.792850 1178 ts_tablet_manager.cc:1397] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.197s user 0.038s sys 0.049s
I20250112 20:51:29.792953 1427 raft_consensus.cc:513] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.793583 1427 leader_election.cc:290] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.793605 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.793711 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64887b3578284cb286ee122ab8c626c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.793699 1178 raft_consensus.cc:357] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.793938 1178 raft_consensus.cc:383] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.794034 1178 raft_consensus.cc:738] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.777179 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 (DEFAULT_TABLE table=default.Table_90937bd663d44575baa256f844890ba9 [id=942d944d07b146fbb1e30e593fa0a266]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.794313 1178 consensus_queue.cc:260] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.794396 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.794943 710 leader_election.cc:343] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.795425 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.795949 710 leader_election.cc:343] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.796074 710 leader_election.cc:304] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.797027 1184 tablet_replica.cc:331] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:29.797133 1178 ts_tablet_manager.cc:1428] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:29.797286 1184 raft_consensus.cc:2238] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.797526 1178 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.797552 1184 pending_rounds.cc:62] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:29.797609 1184 pending_rounds.cc:69] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585006317629440 op_type: NO_OP noop_request { }
I20250112 20:51:29.797752 1184 raft_consensus.cc:2883] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:29.797861 1184 raft_consensus.cc:2267] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.798126 1427 raft_consensus.cc:2743] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.799780 1184 ts_tablet_manager.cc:1905] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.800107 1178 tablet_bootstrap.cc:654] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.800925 1185 log.cc:1198] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/fc63526f932841509277f4726303c801
I20250112 20:51:29.800998 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6d5d425e6a493daa46900cbd8db166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.801529 1185 ts_tablet_manager.cc:1939] T fc63526f932841509277f4726303c801 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:29.805235 1427 raft_consensus.cc:491] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.805366 1427 raft_consensus.cc:513] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.805943 1427 leader_election.cc:290] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:29.806638 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 83c13126e53b46c0a1e7beafe1b0d295 (table default.Table_c62b92e5784e44af9e27e902aa48a6ca [id=212d41ddb2d248a0bcf3aa02f7b10bcd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 83c13126e53b46c0a1e7beafe1b0d295 consensus state has no leader
I20250112 20:51:29.806733 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.807323 710 leader_election.cc:343] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.813098 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.815001 710 leader_election.cc:343] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.815127 710 leader_election.cc:304] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.815649 1427 raft_consensus.cc:2743] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.816555 1164 ts_tablet_manager.cc:1428] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250112 20:51:29.817332 1163 tablet_bootstrap.cc:654] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.817895 1164 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.749261 1186 rpcz_store.cc:276] 0112 20:51:28.619238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.619292 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:29.735139 (+1115847us) ts_tablet_manager.cc:1145] Deleting tablet e0f1c76db0d64624bd43f0e538f0649d
0112 20:51:29.735151 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.739720 (+ 4569us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.742629 (+ 2909us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.749101 (+ 6472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115607}
I20250112 20:51:29.818949 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 200) took 1196 ms. Trace:
I20250112 20:51:29.819115 1186 rpcz_store.cc:276] 0112 20:51:28.622009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.622070 (+ 61us) service_pool.cc:225] Handling call
0112 20:51:29.818794 (+1196724us) ts_tablet_manager.cc:1145] Deleting tablet f249d70b116443b7823e093efeda5cc3
0112 20:51:29.818806 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.818898 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196504}
I20250112 20:51:29.819571 1164 tablet_bootstrap.cc:654] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.819705 1186 tablet_replica.cc:331] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.819970 1186 raft_consensus.cc:2238] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:29.820194 1186 raft_consensus.cc:2267] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.820930 1186 ts_tablet_manager.cc:1905] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.825932 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ed45eb5cbd2e489c83c3d343ca1adcfc (table default.Table_a5ce6ae642344a2c950ae993e6d90311 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:29.826289 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 296703eedc6d459880a74b38e642bceb (table default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]) successfully deleted
I20250112 20:51:29.826969 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d48b17628cf145b69e5fdde4131d9fd2 (table default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]) successfully deleted
W20250112 20:51:29.827303 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b32dc02497f64a2e9ad99da6c9115d5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32dc02497f64a2e9ad99da6c9115d5d already in progress: creating tablet
W20250112 20:51:29.827641 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 00c0712ab2de43eea8c77ab39a971a2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00c0712ab2de43eea8c77ab39a971a2c already in progress: creating tablet
I20250112 20:51:29.827931 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]) successfully deleted
W20250112 20:51:29.828246 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet bc5af0bc83bc415b972cbbe0fb39f75f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc5af0bc83bc415b972cbbe0fb39f75f already in progress: creating tablet
I20250112 20:51:29.837500 1184 ts_tablet_manager.cc:1918] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:29.837844 1184 log.cc:1198] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:29.838402 1184 ts_tablet_manager.cc:1939] T 5ea7336d13ea46748167e35e7e7e8c98 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:29.842033 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f769533bfde0448792d01b883e41763c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f769533bfde0448792d01b883e41763c already in progress: creating tablet
W20250112 20:51:29.842450 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet af1aac31fd32400b9094f8e2de4fe42c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af1aac31fd32400b9094f8e2de4fe42c already in progress: creating tablet
W20250112 20:51:29.842747 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6fe28f23448c47d3979f1982f939e39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fe28f23448c47d3979f1982f939e39c already in progress: creating tablet
I20250112 20:51:29.843899 1163 tablet_bootstrap.cc:492] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.844184 1163 ts_tablet_manager.cc:1397] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.000s
I20250112 20:51:29.845297 1163 raft_consensus.cc:357] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.845666 1163 raft_consensus.cc:383] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.845800 1163 raft_consensus.cc:738] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.846212 1163 consensus_queue.cc:260] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:29.846805 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b84624dc685b401192ca753ff59da016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84624dc685b401192ca753ff59da016 already in progress: creating tablet
I20250112 20:51:29.847689 1163 ts_tablet_manager.cc:1428] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20250112 20:51:29.848048 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8b414be57990432bb786daf5faf0cc2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b414be57990432bb786daf5faf0cc2e already in progress: creating tablet
I20250112 20:51:29.848533 1163 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:29.848866 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d8330fa1e4654a1a9f167a1120a103fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8330fa1e4654a1a9f167a1120a103fc already in progress: creating tablet
W20250112 20:51:29.849189 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 0e2235e2a63c4e1192b3260085908445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2235e2a63c4e1192b3260085908445 already in progress: creating tablet
I20250112 20:51:29.856163 1186 ts_tablet_manager.cc:1918] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:29.851163 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cc971026f6b44db6af8215890ba71075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc971026f6b44db6af8215890ba71075 already in progress: creating tablet
W20250112 20:51:29.857250 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6f4421d16e3f423794c44970a2469b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4421d16e3f423794c44970a2469b10 already in progress: creating tablet
W20250112 20:51:29.858829 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 201a7e2f17bb46fa918dad8f878a4065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 201a7e2f17bb46fa918dad8f878a4065 already in progress: creating tablet
I20250112 20:51:29.859594 1163 tablet_bootstrap.cc:654] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.860280 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e0f1c76db0d64624bd43f0e538f0649d (table default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981 [id=5178ded07b844cf4abc1616c387b8585]) successfully deleted
W20250112 20:51:29.866704 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f249d70b116443b7823e093efeda5cc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f249d70b116443b7823e093efeda5cc3 already in progress: creating tablet
I20250112 20:51:29.867053 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b13d3f3e597f4876bce5852961f37ea0 (table default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]) successfully deleted
W20250112 20:51:29.867355 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7f79b3309e45afb9e18cad97ddfeb0 already in progress: creating tablet
W20250112 20:51:29.867749 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9f0092258e094a38ba7e223be03d4c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f0092258e094a38ba7e223be03d4c9a already in progress: creating tablet
I20250112 20:51:29.868855 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4d616a8ad9ad4777b44deb77bc321d66 (table default.Table_afbd47897b344f1db9916ab7e703ab36 [id=2fd5583e6e114bcdbdbc723643735fd8]) successfully deleted
I20250112 20:51:29.868991 1531 catalog_manager.cc:1260] Loaded cluster ID: 55c7c6228b4d4e3e80a4f1e922af635d
W20250112 20:51:29.869786 440 catalog_manager.cc:4951] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): alter failed for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]): Aborted: Op aborted
I20250112 20:51:29.870015 1164 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.870260 1164 ts_tablet_manager.cc:1397] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.035s
I20250112 20:51:29.870723 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
I20250112 20:51:29.857422 1178 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.871217 1186 log.cc:1198] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:29.871310 1178 ts_tablet_manager.cc:1397] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.074s user 0.025s sys 0.020s
I20250112 20:51:29.871197 1164 raft_consensus.cc:357] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.871451 1164 raft_consensus.cc:383] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.871582 1164 raft_consensus.cc:738] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.871702 1186 ts_tablet_manager.cc:1939] T 5ea7336d13ea46748167e35e7e7e8c98 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:29.872090 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 23d5269a5c8c4c049265e18bff4e2c45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d5269a5c8c4c049265e18bff4e2c45 already in progress: creating tablet
I20250112 20:51:29.871915 1164 consensus_queue.cc:260] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.872745 1164 ts_tablet_manager.cc:1428] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:29.873198 1164 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.875530 1164 tablet_bootstrap.cc:654] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.880848 1531 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: loading cluster ID for follower catalog manager: success
I20250112 20:51:29.883620 1164 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.883891 1164 ts_tablet_manager.cc:1397] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250112 20:51:29.884737 1164 raft_consensus.cc:357] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.885012 1164 raft_consensus.cc:383] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.885154 1164 raft_consensus.cc:738] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.872222 1178 raft_consensus.cc:357] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.885730 1178 raft_consensus.cc:383] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:29.885754 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
I20250112 20:51:29.885839 1178 raft_consensus.cc:738] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.885836 1164 consensus_queue.cc:260] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:29.886108 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
I20250112 20:51:29.886070 1178 consensus_queue.cc:260] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.887113 1178 ts_tablet_manager.cc:1428] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250112 20:51:29.887166 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.887424 1178 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.873453 1436 raft_consensus.cc:491] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.888712 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:29.888919 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
I20250112 20:51:29.888899 1436 raft_consensus.cc:513] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.889089 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]). Expected version 2 got 0
I20250112 20:51:29.889240 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:29.889567 1436 leader_election.cc:290] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.890369 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 201) took 1198 ms. Trace:
I20250112 20:51:29.890463 1186 rpcz_store.cc:276] 0112 20:51:28.692092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.692305 (+ 213us) service_pool.cc:225] Handling call
0112 20:51:29.819607 (+1127302us) ts_tablet_manager.cc:1145] Deleting tablet 5ea7336d13ea46748167e35e7e7e8c98
0112 20:51:29.819618 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:29.851849 (+ 32231us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.856030 (+ 4181us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:29.890309 (+ 34279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127021}
I20250112 20:51:29.892098 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953d2ec8a7564c0bbcf0166487dd1746" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.893055 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
W20250112 20:51:29.893267 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.893361 847 leader_election.cc:304] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:29.893739 1587 raft_consensus.cc:2743] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.894551 1163 tablet_bootstrap.cc:492] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:29.894591 1164 ts_tablet_manager.cc:1428] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250112 20:51:29.894719 1163 ts_tablet_manager.cc:1397] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
I20250112 20:51:29.894865 1178 tablet_bootstrap.cc:654] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.896589 1164 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.895525 1163 raft_consensus.cc:357] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.899137 1164 tablet_bootstrap.cc:654] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.899202 1163 raft_consensus.cc:383] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.899349 1163 raft_consensus.cc:738] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.899672 1163 consensus_queue.cc:260] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.906848 1531 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: acquiring CA information for follower catalog manager: success
I20250112 20:51:29.907429 1163 ts_tablet_manager.cc:1428] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 20:51:29.912158 1178 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.912424 1178 ts_tablet_manager.cc:1397] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.003s
I20250112 20:51:29.913888 1178 raft_consensus.cc:357] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.914175 1178 raft_consensus.cc:383] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.914289 1178 raft_consensus.cc:738] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.914295 1163 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:29.914616 1178 consensus_queue.cc:260] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.915712 1178 ts_tablet_manager.cc:1428] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:29.916651 1163 tablet_bootstrap.cc:654] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.917330 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.917582 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.920058 1164 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.920401 1164 ts_tablet_manager.cc:1397] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20250112 20:51:29.921188 1164 raft_consensus.cc:357] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.921454 1164 raft_consensus.cc:383] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.921562 1164 raft_consensus.cc:738] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.921862 1164 consensus_queue.cc:260] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.922480 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.890849 1186 tablet_replica.cc:331] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:29.923043 1186 raft_consensus.cc:2238] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:29.924389 1164 ts_tablet_manager.cc:1428] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:29.924803 1164 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.928175 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]). Expected version 2 got 0
I20250112 20:51:29.929435 1164 tablet_bootstrap.cc:654] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.930617 1186 raft_consensus.cc:2267] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:29.931911 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.933581 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]). Expected version 2 got 0
I20250112 20:51:29.931320 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.934870 1178 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.936700 1178 tablet_bootstrap.cc:654] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.937536 1164 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:29.937736 1164 ts_tablet_manager.cc:1397] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20250112 20:51:29.938705 1164 raft_consensus.cc:357] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.938982 1164 raft_consensus.cc:383] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.939097 1164 raft_consensus.cc:738] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.939338 1164 consensus_queue.cc:260] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.940145 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.939687 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.941890 1164 ts_tablet_manager.cc:1428] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:29.942204 1164 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.944916 1531 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 20:51:29.946053 1164 tablet_bootstrap.cc:654] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.947515 1427 raft_consensus.cc:491] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.947732 1186 ts_tablet_manager.cc:1905] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:29.947669 1427 raft_consensus.cc:513] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.948307 1427 leader_election.cc:290] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:29.949185 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.950093 710 leader_election.cc:343] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.951336 887 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.952438 753 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.922706 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.938445 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.954975 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:29.949623 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:29.951774 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.954566 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:29.955993 1164 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:29.956100 710 leader_election.cc:343] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.956274 710 leader_election.cc:304] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.956391 1178 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.956512 1178 ts_tablet_manager.cc:1397] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20250112 20:51:29.956697 1427 raft_consensus.cc:2743] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.956254 1164 ts_tablet_manager.cc:1397] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20250112 20:51:29.957428 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:29.957353 1178 raft_consensus.cc:357] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.958127 1178 raft_consensus.cc:383] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.958055 1164 raft_consensus.cc:357] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.958245 1178 raft_consensus.cc:738] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.958303 1164 raft_consensus.cc:383] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.958501 1164 raft_consensus.cc:738] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.958524 1178 consensus_queue.cc:260] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.958788 1164 consensus_queue.cc:260] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:29.959139 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3eeb085e764941198ae7f653bc24757d (DEFAULT_TABLE table=default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.959527 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eeb085e764941198ae7f653bc24757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.960991 1427 raft_consensus.cc:491] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.961119 1427 raft_consensus.cc:513] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:29.961699 1427 leader_election.cc:290] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:29.961889 1178 ts_tablet_manager.cc:1428] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:29.962327 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc971026f6b44db6af8215890ba71075" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:29.963078 710 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.963094 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc971026f6b44db6af8215890ba71075" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:29.963558 710 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:29.963676 710 leader_election.cc:304] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:29.964033 1427 raft_consensus.cc:2743] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:29.964414 1164 ts_tablet_manager.cc:1428] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:29.964495 1178 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:29.964721 1164 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:29.966420 1164 tablet_bootstrap.cc:654] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.967814 1178 tablet_bootstrap.cc:654] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.972203 753 tablet_service.cc:1467] Processing CreateTablet for tablet 3eeb085e764941198ae7f653bc24757d (DEFAULT_TABLE table=default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.972615 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eeb085e764941198ae7f653bc24757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.980020 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3eeb085e764941198ae7f653bc24757d (DEFAULT_TABLE table=default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.980177 1178 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:29.980451 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eeb085e764941198ae7f653bc24757d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.981402 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.981923 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ff8ac6f4d94c10b03a764bbf910c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.985373 890 tablet_service.cc:1467] Processing CreateTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.985814 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ff8ac6f4d94c10b03a764bbf910c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.988996 1178 ts_tablet_manager.cc:1397] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20250112 20:51:29.990294 1178 raft_consensus.cc:357] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.990756 1178 raft_consensus.cc:383] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.990998 1178 raft_consensus.cc:738] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.991451 1178 consensus_queue.cc:260] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:29.997009 1178 ts_tablet_manager.cc:1428] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:29.992954 753 tablet_service.cc:1467] Processing CreateTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:29.998903 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ff8ac6f4d94c10b03a764bbf910c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.999003 1354 raft_consensus.cc:491] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:29.999213 1354 raft_consensus.cc:513] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.000105 1354 leader_election.cc:290] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.001780 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf (DEFAULT_TABLE table=default.Table_21b7926d70e24852a7e66dbb6449e4ae [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.002184 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ebb3c30194d9fa6a092d9bec4eadf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:30.002483 983 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 296703eedc6d459880a74b38e642bceb
W20250112 20:51:30.002946 983 leader_election.cc:343] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 296703eedc6d459880a74b38e642bceb
I20250112 20:51:30.003113 983 leader_election.cc:304] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.005966 1354 raft_consensus.cc:2743] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.013221 887 tablet_service.cc:1467] Processing CreateTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf (DEFAULT_TABLE table=default.Table_21b7926d70e24852a7e66dbb6449e4ae [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.013805 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ebb3c30194d9fa6a092d9bec4eadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.017571 749 tablet_service.cc:1467] Processing CreateTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf (DEFAULT_TABLE table=default.Table_21b7926d70e24852a7e66dbb6449e4ae [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.018078 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ebb3c30194d9fa6a092d9bec4eadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.018306 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbb679585fc44699553fe85ea044566 (DEFAULT_TABLE table=default.Table_ef68d20727c04b09980aac02d08a81bb [id=2bc2a6f29dda4e13a781af77ba66b46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.018620 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb679585fc44699553fe85ea044566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.018568 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbb679585fc44699553fe85ea044566 (DEFAULT_TABLE table=default.Table_ef68d20727c04b09980aac02d08a81bb [id=2bc2a6f29dda4e13a781af77ba66b46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.018949 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb679585fc44699553fe85ea044566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.021355 881 tablet_service.cc:1467] Processing CreateTablet for tablet 35dcd1a120684446bbe19303fb8416ef (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.021739 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35dcd1a120684446bbe19303fb8416ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.024734 752 tablet_service.cc:1467] Processing CreateTablet for tablet 35dcd1a120684446bbe19303fb8416ef (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.025116 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35dcd1a120684446bbe19303fb8416ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.027926 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbb679585fc44699553fe85ea044566 (DEFAULT_TABLE table=default.Table_ef68d20727c04b09980aac02d08a81bb [id=2bc2a6f29dda4e13a781af77ba66b46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.028357 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb679585fc44699553fe85ea044566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.029106 1178 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.033820 1178 tablet_bootstrap.cc:654] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.034518 1164 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.034709 1164 ts_tablet_manager.cc:1397] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.035s
I20250112 20:51:30.036008 1164 raft_consensus.cc:357] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.037117 1164 raft_consensus.cc:383] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.037266 1164 raft_consensus.cc:738] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.037533 1164 consensus_queue.cc:260] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.040443 1164 ts_tablet_manager.cc:1428] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:30.040771 1164 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.042459 1164 tablet_bootstrap.cc:654] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.046283 752 tablet_service.cc:1467] Processing CreateTablet for tablet d570daec845f40778c48ff3f2e2e30cb (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.046941 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d570daec845f40778c48ff3f2e2e30cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.046988 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 35dcd1a120684446bbe19303fb8416ef (DEFAULT_TABLE table=default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.047245 1178 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.047387 1178 ts_tablet_manager.cc:1397] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250112 20:51:30.048242 1178 raft_consensus.cc:357] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.048449 1178 raft_consensus.cc:383] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.048537 1178 raft_consensus.cc:738] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.048820 1178 consensus_queue.cc:260] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.049458 1178 ts_tablet_manager.cc:1428] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.053102 1178 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.054955 1178 tablet_bootstrap.cc:654] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.059446 1427 raft_consensus.cc:491] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.059605 1427 raft_consensus.cc:513] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.060523 1427 leader_election.cc:290] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.061373 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.061559 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.062204 710 leader_election.cc:343] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.062481 710 leader_election.cc:343] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.062633 710 leader_election.cc:304] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.063128 1427 raft_consensus.cc:2743] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.065310 1587 raft_consensus.cc:491] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.065507 1587 raft_consensus.cc:513] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.066398 1587 leader_election.cc:290] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.067754 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d3f3e597f4876bce5852961f37ea0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.069342 847 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.069633 1018 tablet_service.cc:1467] Processing CreateTablet for tablet d570daec845f40778c48ff3f2e2e30cb (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.070153 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d570daec845f40778c48ff3f2e2e30cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.071831 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 36935e6d8c5643388ad163458e2b387c (DEFAULT_TABLE table=default.Table_53ad9b3fe99c4e67ad2569718afe2e16 [id=91351703475c45e69c55a3ee47c4bb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.072258 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36935e6d8c5643388ad163458e2b387c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:30.073186 847 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:30.073366 847 leader_election.cc:304] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.073964 751 tablet_service.cc:1467] Processing CreateTablet for tablet 36935e6d8c5643388ad163458e2b387c (DEFAULT_TABLE table=default.Table_53ad9b3fe99c4e67ad2569718afe2e16 [id=91351703475c45e69c55a3ee47c4bb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.074425 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36935e6d8c5643388ad163458e2b387c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.074498 882 tablet_service.cc:1467] Processing CreateTablet for tablet 36935e6d8c5643388ad163458e2b387c (DEFAULT_TABLE table=default.Table_53ad9b3fe99c4e67ad2569718afe2e16 [id=91351703475c45e69c55a3ee47c4bb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.075035 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36935e6d8c5643388ad163458e2b387c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.077021 1587 raft_consensus.cc:2743] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.082381 881 tablet_service.cc:1467] Processing CreateTablet for tablet d570daec845f40778c48ff3f2e2e30cb (DEFAULT_TABLE table=default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.082934 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d570daec845f40778c48ff3f2e2e30cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.086352 1427 raft_consensus.cc:491] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.086488 1427 raft_consensus.cc:513] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.087162 1427 leader_election.cc:290] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.087925 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.088768 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.089442 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2469127891524be5ac3dc13fbcf35c7a (DEFAULT_TABLE table=default.Table_c3afc9f0dcf0422fa7cdd946feec0e58 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.089855 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2469127891524be5ac3dc13fbcf35c7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:30.089812 710 leader_election.cc:343] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.090065 710 leader_election.cc:343] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.090174 710 leader_election.cc:304] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.090644 1427 raft_consensus.cc:2743] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.093174 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2469127891524be5ac3dc13fbcf35c7a (DEFAULT_TABLE table=default.Table_c3afc9f0dcf0422fa7cdd946feec0e58 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.093658 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2469127891524be5ac3dc13fbcf35c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.099365 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:30.099607 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2469127891524be5ac3dc13fbcf35c7a (DEFAULT_TABLE table=default.Table_c3afc9f0dcf0422fa7cdd946feec0e58 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.100306 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2469127891524be5ac3dc13fbcf35c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.101742 1354 raft_consensus.cc:491] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.101938 1354 raft_consensus.cc:513] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.102676 1354 leader_election.cc:290] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:30.103994 983 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:30.104198 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:30.105016 983 leader_election.cc:343] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:30.105134 983 leader_election.cc:304] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.105595 1354 raft_consensus.cc:2743] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.106587 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e040383f6e2a4ccf9614fd2d119a3cb4 (table default.Table_4c35fe9b46fd41b78ea53fd616abc99b [id=19853915fa604539b91aab6a4dc926e1]) successfully deleted
W20250112 20:51:30.106856 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet bc5af0bc83bc415b972cbbe0fb39f75f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc5af0bc83bc415b972cbbe0fb39f75f already in progress: creating tablet
I20250112 20:51:30.107179 446 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0]). Expected version 1 got 0
I20250112 20:51:30.107316 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b310e4135c2b424fb006a38a60e41066 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.107455 446 catalog_manager.cc:5526] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:30.107915 446 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
I20250112 20:51:30.108613 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b437c31235f14edb94ca1e295cb9487e" } new_table_name: "default.Table_70e7a6a54ee6434988d6cfb21ade1f02" modify_external_catalogs: true
I20250112 20:51:30.109844 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8284ddd01ebb44d49bfeac291415f6b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.110388 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8284ddd01ebb44d49bfeac291415f6b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.111549 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_31a6df3a24c042a38ee31d789e5f46eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.112027 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31a6df3a24c042a38ee31d789e5f46eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.113037 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ee162ce42a6b405ba3a6a8e2dbc34d73" } new_table_name: "default.Table_b0787900a74e4ae79083035c64b84c00" modify_external_catalogs: true
I20250112 20:51:30.114087 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c028f60e96ab4c71bc0e9e3dabf2bcd0" } new_table_name: "default.Table_14ab0d3d40e14fdfaec2dfe76fd50455" modify_external_catalogs: true
I20250112 20:51:30.114977 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 31d3669355bc4b6bbc665edaa80c7a9e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.115970 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_211ca26c30e94a03aaf75698f6765585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.116389 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_211ca26c30e94a03aaf75698f6765585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.117374 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1b4f716257454c0bbc3266a5706d97e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.117748 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b4f716257454c0bbc3266a5706d97e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.121601 1163 tablet_bootstrap.cc:492] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.121752 1163 ts_tablet_manager.cc:1397] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.208s user 0.000s sys 0.004s
I20250112 20:51:30.122705 1354 raft_consensus.cc:491] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.122891 1354 raft_consensus.cc:513] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.123539 1354 leader_election.cc:290] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.124027 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35dcd1a120684446bbe19303fb8416ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.122932 1163 raft_consensus.cc:357] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.125258 1163 raft_consensus.cc:383] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.125352 1163 raft_consensus.cc:738] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.125598 1163 consensus_queue.cc:260] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.127221 1163 ts_tablet_manager.cc:1428] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:51:30.127566 1163 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.129298 1163 tablet_bootstrap.cc:654] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.132567 1186 ts_tablet_manager.cc:1918] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:30.132726 1186 log.cc:1198] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e7a8468369af418ea4a1b0a681f8ff74
W20250112 20:51:30.135471 983 leader_election.cc:343] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: cc9aa4f42f89455cba7c72e7a49c6a94
W20250112 20:51:30.141111 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet af1aac31fd32400b9094f8e2de4fe42c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af1aac31fd32400b9094f8e2de4fe42c already in progress: creating tablet
I20250112 20:51:30.143563 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e7a8468369af418ea4a1b0a681f8ff74 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:30.142072 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f769533bfde0448792d01b883e41763c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f769533bfde0448792d01b883e41763c already in progress: creating tablet
W20250112 20:51:30.146972 983 leader_election.cc:343] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:30.147135 983 leader_election.cc:304] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:30.147203 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6fe28f23448c47d3979f1982f939e39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fe28f23448c47d3979f1982f939e39c already in progress: creating tablet
W20250112 20:51:30.147423 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 0e2235e2a63c4e1192b3260085908445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e2235e2a63c4e1192b3260085908445 already in progress: creating tablet
I20250112 20:51:30.147733 1354 raft_consensus.cc:2743] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:30.147764 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b84624dc685b401192ca753ff59da016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84624dc685b401192ca753ff59da016 already in progress: creating tablet
I20250112 20:51:30.147941 1354 consensus_queue.cc:1035] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:30.148068 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8b414be57990432bb786daf5faf0cc2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b414be57990432bb786daf5faf0cc2e already in progress: creating tablet
W20250112 20:51:30.148293 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d8330fa1e4654a1a9f167a1120a103fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8330fa1e4654a1a9f167a1120a103fc already in progress: creating tablet
W20250112 20:51:30.148481 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cc971026f6b44db6af8215890ba71075 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc971026f6b44db6af8215890ba71075 already in progress: creating tablet
W20250112 20:51:30.148757 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6f4421d16e3f423794c44970a2469b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4421d16e3f423794c44970a2469b10 already in progress: creating tablet
W20250112 20:51:30.149036 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 201a7e2f17bb46fa918dad8f878a4065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 201a7e2f17bb46fa918dad8f878a4065 already in progress: creating tablet
W20250112 20:51:30.149245 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e0f1c76db0d64624bd43f0e538f0649d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f1c76db0d64624bd43f0e538f0649d already in progress: creating tablet
W20250112 20:51:30.149427 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f249d70b116443b7823e093efeda5cc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f249d70b116443b7823e093efeda5cc3 already in progress: creating tablet
W20250112 20:51:30.149649 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet eda44c48bf9746b59ccdc4d39a41ea91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eda44c48bf9746b59ccdc4d39a41ea91 already in progress: creating tablet
I20250112 20:51:30.149813 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 719b906973a8490898d6befce49910d3 (table default.Table_54895a2507b64553b62c2b8dbe48a747 [id=d0314442c0ce488baf64ef462029316b]) successfully deleted
W20250112 20:51:30.150038 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3c14102aecb743e2a82660039bd0f038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c14102aecb743e2a82660039bd0f038 already in progress: creating tablet
W20250112 20:51:30.150213 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7f79b3309e45afb9e18cad97ddfeb0 already in progress: creating tablet
W20250112 20:51:30.150431 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9f0092258e094a38ba7e223be03d4c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f0092258e094a38ba7e223be03d4c9a already in progress: creating tablet
I20250112 20:51:30.150609 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4d616a8ad9ad4777b44deb77bc321d66 (table default.Table_afbd47897b344f1db9916ab7e703ab36 [id=2fd5583e6e114bcdbdbc723643735fd8]) successfully deleted
I20250112 20:51:30.151074 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
W20250112 20:51:30.151268 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 23d5269a5c8c4c049265e18bff4e2c45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d5269a5c8c4c049265e18bff4e2c45 already in progress: creating tablet
W20250112 20:51:30.151458 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
W20250112 20:51:30.151609 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
I20250112 20:51:30.152045 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5ea7336d13ea46748167e35e7e7e8c98 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]) successfully deleted
W20250112 20:51:30.152232 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 64cd83b199bb4da28d39da449945c389 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cd83b199bb4da28d39da449945c389 already in progress: creating tablet
W20250112 20:51:30.152429 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 already in progress: creating tablet
W20250112 20:51:30.152628 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2b8250621cbe45eba669b5ef14e672c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b8250621cbe45eba669b5ef14e672c2 already in progress: creating tablet
I20250112 20:51:30.155084 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ff1653ec182d4a2bbdd3c2a01b91d268" } modify_external_catalogs: true
I20250112 20:51:30.155465 1436 raft_consensus.cc:491] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.155505 1178 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.155525 1186 ts_tablet_manager.cc:1939] T e7a8468369af418ea4a1b0a681f8ff74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.155647 1436 raft_consensus.cc:513] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.155737 1178 ts_tablet_manager.cc:1397] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.103s user 0.017s sys 0.028s
I20250112 20:51:30.156634 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 202) took 1464 ms. Trace:
I20250112 20:51:30.156752 1186 rpcz_store.cc:276] 0112 20:51:28.692245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.692626 (+ 381us) service_pool.cc:225] Handling call
0112 20:51:29.890775 (+1198149us) ts_tablet_manager.cc:1145] Deleting tablet e7a8468369af418ea4a1b0a681f8ff74
0112 20:51:29.890784 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:30.082513 (+191729us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:30.132390 (+ 49877us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:30.156593 (+ 24203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197905}
I20250112 20:51:30.155694 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86c60a78bd264bc08c315269fdf3240f" } modify_external_catalogs: true
I20250112 20:51:30.157073 1436 leader_election.cc:290] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.157178 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86c60a78bd264bc08c315269fdf3240f" } modify_external_catalogs: true
I20250112 20:51:30.157251 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 210) took 1242 ms. Trace:
I20250112 20:51:30.157181 1178 raft_consensus.cc:357] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.157410 1178 raft_consensus.cc:383] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.157505 1178 raft_consensus.cc:738] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.157745 1612 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:30.157742 1178 consensus_queue.cc:260] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.158228 1178 ts_tablet_manager.cc:1428] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.158555 1178 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:30.158521 847 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:30.159999 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b70ffb3e3341d398d36ff76993e9ac" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.160722 1178 tablet_bootstrap.cc:654] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.156380 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_14bd18a76fc94d41b11ad4252e4c491f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.161146 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
W20250112 20:51:30.161484 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14bd18a76fc94d41b11ad4252e4c491f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:30.161916 847 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.162055 847 leader_election.cc:304] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.162652 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ff1653ec182d4a2bbdd3c2a01b91d268" } modify_external_catalogs: true
I20250112 20:51:30.164268 1587 raft_consensus.cc:2743] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.168542 1163 tablet_bootstrap.cc:492] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.168738 1163 ts_tablet_manager.cc:1397] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250112 20:51:30.169847 1163 raft_consensus.cc:357] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.170202 1163 raft_consensus.cc:383] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.170344 1163 raft_consensus.cc:738] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.157347 1186 rpcz_store.cc:276] 0112 20:51:28.915248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:28.916558 (+ 1310us) service_pool.cc:225] Handling call
0112 20:51:30.157052 (+1240494us) ts_tablet_manager.cc:1145] Deleting tablet eda44c48bf9746b59ccdc4d39a41ea91
0112 20:51:30.157065 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:30.157211 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238310}
I20250112 20:51:30.170639 1163 consensus_queue.cc:260] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.171345 1163 ts_tablet_manager.cc:1428] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.171666 1163 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.172446 1186 tablet_replica.cc:331] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.172958 1186 raft_consensus.cc:2238] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.173250 1610 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:30.173434 1186 raft_consensus.cc:2267] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.174193 1186 ts_tablet_manager.cc:1905] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.179872 1163 tablet_bootstrap.cc:654] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:30.184885 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet bcbbefa7b55041a6a6b50b3c96c45833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcbbefa7b55041a6a6b50b3c96c45833 already in progress: creating tablet
W20250112 20:51:30.185454 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b4ad7cf1e0c74265a4d21931279f2318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ad7cf1e0c74265a4d21931279f2318 already in progress: creating tablet
I20250112 20:51:30.185673 1186 ts_tablet_manager.cc:1918] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.194530 751 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.194967 1185 tablet_replica.cc:331] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:30.195261 1185 raft_consensus.cc:2238] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.195461 1185 raft_consensus.cc:2267] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.188310 1436 raft_consensus.cc:491] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.195787 1186 log.cc:1198] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4d616a8ad9ad4777b44deb77bc321d66
I20250112 20:51:30.195834 1436 raft_consensus.cc:513] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.196105 1185 ts_tablet_manager.cc:1905] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.196151 1186 ts_tablet_manager.cc:1939] T 4d616a8ad9ad4777b44deb77bc321d66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.196563 1436 leader_election.cc:290] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.196764 1436 raft_consensus.cc:491] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.196823 1186 tablet_replica.cc:331] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.196916 1436 raft_consensus.cc:513] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.197036 1186 raft_consensus.cc:2238] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.197216 1186 pending_rounds.cc:62] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:30.197268 1186 pending_rounds.cc:69] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113585006422933504 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "fc63526f932841509277f4726303c801" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" new_extra_config { } }
I20250112 20:51:30.197510 1436 leader_election.cc:290] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.197750 1186 raft_consensus.cc:2267] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.198846 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.192083 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
I20250112 20:51:30.199258 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.199822 847 leader_election.cc:343] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.200047 847 leader_election.cc:343] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.200726 1186 ts_tablet_manager.cc:1905] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.201126 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.201396 770 raft_consensus.cc:2463] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.201808 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.202050 773 raft_consensus.cc:2463] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.202901 847 leader_election.cc:304] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.203164 1178 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.203296 1178 ts_tablet_manager.cc:1397] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.005s
I20250112 20:51:30.203330 847 leader_election.cc:304] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.203732 1587 raft_consensus.cc:2798] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.203820 1587 raft_consensus.cc:491] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.203917 1587 raft_consensus.cc:3054] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.204164 1178 raft_consensus.cc:357] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.204363 1178 raft_consensus.cc:383] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.204465 1178 raft_consensus.cc:738] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.204921 1185 ts_tablet_manager.cc:1918] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.204833 1178 consensus_queue.cc:260] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.205052 1185 log.cc:1198] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/cc971026f6b44db6af8215890ba71075
I20250112 20:51:30.205435 1185 ts_tablet_manager.cc:1939] T cc971026f6b44db6af8215890ba71075 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:30.205430 1178 ts_tablet_manager.cc:1428] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.205729 1178 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.207078 1587 raft_consensus.cc:513] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.207376 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.207697 1587 leader_election.cc:290] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.207863 1587 raft_consensus.cc:2798] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.207973 1587 raft_consensus.cc:491] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.208084 1587 raft_consensus.cc:3054] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.208887 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:30.209537 847 leader_election.cc:343] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.210107 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.210309 890 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.210340 1436 raft_consensus.cc:491] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.210485 1436 raft_consensus.cc:513] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.210625 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a8468369af418ea4a1b0a681f8ff74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.210626 1587 raft_consensus.cc:513] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.210879 1184 tablet_replica.cc:331] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.211022 1436 leader_election.cc:290] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.211081 1184 raft_consensus.cc:2238] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.211216 1587 leader_election.cc:290] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.211273 1184 raft_consensus.cc:2267] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.211988 1184 ts_tablet_manager.cc:1905] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.212023 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.212378 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616b8aa8fd62486c978c275712064f77" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.212937 847 leader_election.cc:343] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.213160 847 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.213574 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.213778 773 raft_consensus.cc:3054] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.216348 773 raft_consensus.cc:2463] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:30.218497 1184 ts_tablet_manager.cc:1918] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:30.218680 1184 log.cc:1198] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e7a8468369af418ea4a1b0a681f8ff74
I20250112 20:51:30.219693 1184 ts_tablet_manager.cc:1939] T e7a8468369af418ea4a1b0a681f8ff74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.221134 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9aa4f42f89455cba7c72e7a49c6a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.221427 1184 tablet_replica.cc:331] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.221660 1184 raft_consensus.cc:2238] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.221870 1184 raft_consensus.cc:2267] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.222404 1184 ts_tablet_manager.cc:1905] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.223886 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2afa5b0717df4f83a2b44efac12fc06b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.224097 770 raft_consensus.cc:3054] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.225265 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 296703eedc6d459880a74b38e642bceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.225744 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ed45eb5cbd2e489c83c3d343ca1adcfc with delete_type TABLET_DATA_DELETED (replaced by tablet e72fda1743fc4f96b29add6e36e884de) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.225796 1186 ts_tablet_manager.cc:1918] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:30.225898 1186 log.cc:1198] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/fc63526f932841509277f4726303c801
I20250112 20:51:30.226245 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea7336d13ea46748167e35e7e7e8c98 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a90f801d854d4d8f7f09112e2d2929) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.226311 1186 ts_tablet_manager.cc:1939] T fc63526f932841509277f4726303c801 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.226694 770 raft_consensus.cc:2463] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:30.227077 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 64cd83b199bb4da28d39da449945c389 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cd83b199bb4da28d39da449945c389 already in progress: creating tablet
W20250112 20:51:30.227341 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 already in progress: creating tablet
W20250112 20:51:30.227574 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2b8250621cbe45eba669b5ef14e672c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b8250621cbe45eba669b5ef14e672c2 already in progress: creating tablet
I20250112 20:51:30.227785 1186 tablet_replica.cc:331] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:30.227790 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bcbbefa7b55041a6a6b50b3c96c45833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcbbefa7b55041a6a6b50b3c96c45833 already in progress: creating tablet
I20250112 20:51:30.227936 1178 tablet_bootstrap.cc:654] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.227970 847 leader_election.cc:304] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:30.228029 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b4ad7cf1e0c74265a4d21931279f2318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ad7cf1e0c74265a4d21931279f2318 already in progress: creating tablet
I20250112 20:51:30.228879 847 leader_election.cc:304] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.229336 1587 raft_consensus.cc:2798] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:30.229466 847 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 616b8aa8fd62486c978c275712064f77
I20250112 20:51:30.229486 1587 raft_consensus.cc:695] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.229759 1430 raft_consensus.cc:2798] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.229874 1587 consensus_queue.cc:237] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.230948 1186 raft_consensus.cc:2238] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.231772 847 leader_election.cc:304] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.238324 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet cc971026f6b44db6af8215890ba71075 (table default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]) successfully deleted
I20250112 20:51:30.238585 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5ea7336d13ea46748167e35e7e7e8c98 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]) successfully deleted
I20250112 20:51:30.239828 1178 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.239989 1178 ts_tablet_manager.cc:1397] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20250112 20:51:30.240270 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e7a8468369af418ea4a1b0a681f8ff74 (table default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]) successfully deleted
W20250112 20:51:30.240479 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet eda44c48bf9746b59ccdc4d39a41ea91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eda44c48bf9746b59ccdc4d39a41ea91 already in progress: creating tablet
W20250112 20:51:30.240681 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3c14102aecb743e2a82660039bd0f038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c14102aecb743e2a82660039bd0f038 already in progress: creating tablet
W20250112 20:51:30.240957 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7f79b3309e45afb9e18cad97ddfeb0 already in progress: creating tablet
W20250112 20:51:30.241178 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9f0092258e094a38ba7e223be03d4c9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f0092258e094a38ba7e223be03d4c9a already in progress: creating tablet
I20250112 20:51:30.241433 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4d616a8ad9ad4777b44deb77bc321d66 (table default.Table_afbd47897b344f1db9916ab7e703ab36 [id=2fd5583e6e114bcdbdbc723643735fd8]) successfully deleted
I20250112 20:51:30.241691 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
W20250112 20:51:30.241904 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 23d5269a5c8c4c049265e18bff4e2c45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d5269a5c8c4c049265e18bff4e2c45 already in progress: creating tablet
W20250112 20:51:30.242115 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
W20250112 20:51:30.242321 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5d384628ba9b4f81b768eda4f796632b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d384628ba9b4f81b768eda4f796632b already in progress: creating tablet
W20250112 20:51:30.242532 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 64cd83b199bb4da28d39da449945c389 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64cd83b199bb4da28d39da449945c389 already in progress: creating tablet
I20250112 20:51:30.243989 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e7a8468369af418ea4a1b0a681f8ff74 (table default.Table_3d5bec5466fb4f71bbe9b86ec73f874f [id=1c6e879afc3e4a89a5e01307aba6c3df]) successfully deleted
W20250112 20:51:30.255311 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44f11304909b416aa1e9db1abf633a89 consensus state has no leader
W20250112 20:51:30.256610 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3facda2ee004b2e9487194ba040c9cf consensus state has no leader
I20250112 20:51:30.247043 1184 ts_tablet_manager.cc:1918] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.257063 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 31d3669355bc4b6bbc665edaa80c7a9e
I20250112 20:51:30.257194 1184 log.cc:1198] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/cc9aa4f42f89455cba7c72e7a49c6a94
I20250112 20:51:30.257565 449 catalog_manager.cc:6596] ReplaceTablet: tablet 31d3669355bc4b6bbc665edaa80c7a9e deleted and replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:30.257591 1184 ts_tablet_manager.cc:1939] T cc9aa4f42f89455cba7c72e7a49c6a94 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.269152 1186 raft_consensus.cc:2267] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.269946 1186 ts_tablet_manager.cc:1905] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:30.272882 847 raft_consensus.cc:2681] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:30.272986 1184 tablet_replica.cc:331] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:30.273106 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 43640.3us
I20250112 20:51:30.273190 1184 raft_consensus.cc:2238] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.273943 1184 raft_consensus.cc:2267] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.273919 1178 raft_consensus.cc:357] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.274202 1178 raft_consensus.cc:383] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.274284 1178 raft_consensus.cc:738] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.274556 1178 consensus_queue.cc:260] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.275213 1178 ts_tablet_manager.cc:1428] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250112 20:51:30.275585 1178 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.277806 1178 tablet_bootstrap.cc:654] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.269145 1430 raft_consensus.cc:695] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.279156 1430 consensus_queue.cc:237] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.281222 1186 ts_tablet_manager.cc:1918] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.281378 1186 log.cc:1198] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/616b8aa8fd62486c978c275712064f77
I20250112 20:51:30.281762 1186 ts_tablet_manager.cc:1939] T 616b8aa8fd62486c978c275712064f77 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.283499 1186 tablet_replica.cc:331] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.283751 1186 raft_consensus.cc:2238] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.283962 1186 raft_consensus.cc:2267] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.284817 1186 ts_tablet_manager.cc:1905] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.287460 773 raft_consensus.cc:1270] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:30.288132 1587 consensus_queue.cc:1035] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:30.288503 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b310e4135c2b424fb006a38a60e41066
I20250112 20:51:30.289089 453 catalog_manager.cc:6596] ReplaceTablet: tablet b310e4135c2b424fb006a38a60e41066 deleted and replaced by tablet aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:30.294802 1610 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:30.296480 1609 mvcc.cc:204] Tried to move back new op lower bound from 7113585009808502784 to 7113585007360266240. Current Snapshot: MvccSnapshot[applied={T|T < 7113585009808502784}]
I20250112 20:51:30.286821 1044 raft_consensus.cc:3054] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.296957 1626 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:30.297427 1184 ts_tablet_manager.cc:1905] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.297909 1622 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:30.299829 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.293938 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.301604 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.301867 1044 raft_consensus.cc:1270] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:30.304747 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.305612 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 616b8aa8fd62486c978c275712064f77 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
W20250112 20:51:30.305967 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 already in progress: creating tablet
W20250112 20:51:30.306231 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2b8250621cbe45eba669b5ef14e672c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b8250621cbe45eba669b5ef14e672c2 already in progress: creating tablet
W20250112 20:51:30.306463 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet bcbbefa7b55041a6a6b50b3c96c45833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcbbefa7b55041a6a6b50b3c96c45833 already in progress: creating tablet
W20250112 20:51:30.306663 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b4ad7cf1e0c74265a4d21931279f2318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ad7cf1e0c74265a4d21931279f2318 already in progress: creating tablet
I20250112 20:51:30.306900 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet cc9aa4f42f89455cba7c72e7a49c6a94 (table default.Table_9cf0195d130e459fbc8e651f57546be7 [id=7bde50a83284433daf462fe09db9061c]) successfully deleted
W20250112 20:51:30.307137 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 31d3669355bc4b6bbc665edaa80c7a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31d3669355bc4b6bbc665edaa80c7a9e already in progress: creating tablet
W20250112 20:51:30.307376 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b310e4135c2b424fb006a38a60e41066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b310e4135c2b424fb006a38a60e41066 already in progress: creating tablet
I20250112 20:51:30.310384 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_66f53100719d424aa91c8c1ddad82c8a" } new_table_name: "default.Table_d9d198758a4a4d4f83b2a03af392409a" modify_external_catalogs: true
I20250112 20:51:30.313987 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ca9e0810a7294c4fab5de64b7fde63af" } modify_external_catalogs: true
I20250112 20:51:30.314204 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ca9e0810a7294c4fab5de64b7fde63af" } modify_external_catalogs: true
I20250112 20:51:30.322014 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 4390c60d6c724190a216ef9c961b9592 (table default.Table_9c771feed27441ec8340f7279a4a06c1 [id=2ee1f97fef0f4e2b83932132d0936504]). Expected version 2 got 0
I20250112 20:51:30.325035 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eece5bebd4c54ee784339d7cb43daf6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.325713 1436 consensus_queue.cc:1035] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
W20250112 20:51:30.325856 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eece5bebd4c54ee784339d7cb43daf6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.329479 1612 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:30.330054 1579 mvcc.cc:204] Tried to move back new op lower bound from 7113585009808502784 to 7113585007360266240. Current Snapshot: MvccSnapshot[applied={T|T < 7113585009808502784}]
I20250112 20:51:30.334205 1184 ts_tablet_manager.cc:1918] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.340639 1587 raft_consensus.cc:491] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.340816 1587 raft_consensus.cc:513] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.341719 1587 leader_election.cc:290] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.343734 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:30.344152 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81b932b7e1e34d4ab18aae40047dc038 consensus state has no leader
W20250112 20:51:30.344138 847 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: cc971026f6b44db6af8215890ba71075
I20250112 20:51:30.344388 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.344733 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ef05483d74ba466ea37bf2e84489fc15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.345463 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef05483d74ba466ea37bf2e84489fc15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.342690 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc971026f6b44db6af8215890ba71075" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.346124 1184 log.cc:1198] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/296703eedc6d459880a74b38e642bceb
I20250112 20:51:30.346786 1184 ts_tablet_manager.cc:1939] T 296703eedc6d459880a74b38e642bceb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:30.347110 847 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.347223 847 leader_election.cc:304] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.347667 1184 tablet_replica.cc:331] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.348956 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 296703eedc6d459880a74b38e642bceb (table default.Table_fb2be6ce07824fe6a27363d4d6ff0106 [id=12e652705d97445e8fd6b3d4a04d3c7d]) successfully deleted
I20250112 20:51:30.351312 1587 raft_consensus.cc:2743] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.353643 1178 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.354826 1184 raft_consensus.cc:2238] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.355242 1178 ts_tablet_manager.cc:1397] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.010s
I20250112 20:51:30.355329 1184 raft_consensus.cc:2267] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.356096 1184 ts_tablet_manager.cc:1905] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.356087 1178 raft_consensus.cc:357] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.362165 1178 raft_consensus.cc:383] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.362306 1178 raft_consensus.cc:738] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.362556 1178 consensus_queue.cc:260] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.363157 1178 ts_tablet_manager.cc:1428] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:30.363543 1178 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.366674 1178 tablet_bootstrap.cc:654] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.362058 1186 ts_tablet_manager.cc:1918] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.375451 1186 log.cc:1198] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9a692cc5fb6f4afc854c4821274edfdb
I20250112 20:51:30.375906 1186 ts_tablet_manager.cc:1939] T 9a692cc5fb6f4afc854c4821274edfdb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.376089 1184 ts_tablet_manager.cc:1918] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.377005 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9a692cc5fb6f4afc854c4821274edfdb (table default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]) successfully deleted
I20250112 20:51:30.377399 1186 tablet_replica.cc:331] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.377641 1186 raft_consensus.cc:2238] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.377857 1186 raft_consensus.cc:2267] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.378525 1186 ts_tablet_manager.cc:1905] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.377031 1163 tablet_bootstrap.cc:492] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:30.381448 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 233874us
I20250112 20:51:30.385102 1163 ts_tablet_manager.cc:1397] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.213s user 0.012s sys 0.015s
I20250112 20:51:30.386266 1186 ts_tablet_manager.cc:1918] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.386337 1163 raft_consensus.cc:357] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.386586 1163 raft_consensus.cc:383] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.387250 1163 raft_consensus.cc:738] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.387548 1163 consensus_queue.cc:260] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.389232 1163 ts_tablet_manager.cc:1428] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:30.389572 1163 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.389604 1354 consensus_queue.cc:1035] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:30.391217 1163 tablet_bootstrap.cc:654] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.391917 1186 log.cc:1198] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:30.391379 1184 log.cc:1198] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ed45eb5cbd2e489c83c3d343ca1adcfc
I20250112 20:51:30.392549 1184 ts_tablet_manager.cc:1939] T ed45eb5cbd2e489c83c3d343ca1adcfc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.392549 1186 ts_tablet_manager.cc:1939] T 29b70ffb3e3341d398d36ff76993e9ac P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.393360 1178 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.393501 1164 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.393549 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ed45eb5cbd2e489c83c3d343ca1adcfc (table default.Table_a5ce6ae642344a2c950ae993e6d90311 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:30.393690 1164 ts_tablet_manager.cc:1397] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.353s user 0.064s sys 0.063s
W20250112 20:51:30.393889 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5ea7336d13ea46748167e35e7e7e8c98 because the tablet was not found. No further retry: Not found: Tablet not found: 5ea7336d13ea46748167e35e7e7e8c98
I20250112 20:51:30.393903 1427 raft_consensus.cc:491] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.394055 1427 raft_consensus.cc:513] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:30.394165 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 31d3669355bc4b6bbc665edaa80c7a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31d3669355bc4b6bbc665edaa80c7a9e already in progress: creating tablet
W20250112 20:51:30.394454 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b310e4135c2b424fb006a38a60e41066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b310e4135c2b424fb006a38a60e41066 already in progress: creating tablet
I20250112 20:51:30.394835 1427 leader_election.cc:290] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.394908 1178 ts_tablet_manager.cc:1397] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20250112 20:51:30.395828 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.396185 1178 raft_consensus.cc:357] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:30.396414 710 leader_election.cc:343] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.396456 1178 raft_consensus.cc:383] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.396605 1178 raft_consensus.cc:738] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.396989 1178 consensus_queue.cc:260] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.397989 1178 ts_tablet_manager.cc:1428] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:30.398488 1164 raft_consensus.cc:357] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.398708 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:30.399425 710 leader_election.cc:343] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.399574 710 leader_election.cc:304] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.399997 1427 raft_consensus.cc:2743] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.400650 1178 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.399415 1164 raft_consensus.cc:383] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.402297 1164 raft_consensus.cc:738] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.402851 1164 consensus_queue.cc:260] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.402997 1178 tablet_bootstrap.cc:654] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.404081 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 29b70ffb3e3341d398d36ff76993e9ac (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
I20250112 20:51:30.404500 1186 tablet_replica.cc:331] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.404538 1164 ts_tablet_manager.cc:1428] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250112 20:51:30.404837 1186 raft_consensus.cc:2238] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.405121 1186 raft_consensus.cc:2267] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.405247 1164 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.406104 1186 ts_tablet_manager.cc:1905] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.407138 1164 tablet_bootstrap.cc:654] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.416110 1427 raft_consensus.cc:491] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.416251 1427 raft_consensus.cc:513] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.416898 1427 leader_election.cc:290] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.417991 1163 tablet_bootstrap.cc:492] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.418203 1163 ts_tablet_manager.cc:1397] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20250112 20:51:30.418269 1178 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.418452 1178 ts_tablet_manager.cc:1397] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250112 20:51:30.418151 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.418388 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.418916 911 raft_consensus.cc:2463] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:30.419095 1163 raft_consensus.cc:357] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.419306 710 leader_election.cc:304] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:30.419369 1163 raft_consensus.cc:383] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:30.419589 710 leader_election.cc:343] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.419531 1178 raft_consensus.cc:357] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.419744 1178 raft_consensus.cc:383] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.419855 1178 raft_consensus.cc:738] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.419839 1163 raft_consensus.cc:738] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.420068 1427 raft_consensus.cc:2798] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.420178 1427 raft_consensus.cc:491] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.420096 1163 consensus_queue.cc:260] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.420257 1427 raft_consensus.cc:3054] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.420096 1178 consensus_queue.cc:260] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.420748 1178 ts_tablet_manager.cc:1428] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:30.420835 1163 ts_tablet_manager.cc:1428] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:30.421097 1178 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.421097 1163 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.422714 1178 tablet_bootstrap.cc:654] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.422725 1163 tablet_bootstrap.cc:654] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.422850 1427 raft_consensus.cc:513] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.424364 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:30.424846 710 leader_election.cc:343] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.425602 1427 leader_election.cc:290] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.427348 1186 ts_tablet_manager.cc:1918] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.428030 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:30.428930 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_62d64f27a517493eb41fe5974ca444ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.429509 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d64f27a517493eb41fe5974ca444ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.431423 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15613426e60a4f298722927f651c546d" } modify_external_catalogs: true
I20250112 20:51:30.431620 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15613426e60a4f298722927f651c546d" } modify_external_catalogs: true
W20250112 20:51:30.432080 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85897bfec1834c57956a23c5dfc2e853 consensus state has no leader
I20250112 20:51:30.432417 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eee58a17649b4263951fabefc0a4dfbf" } new_table_name: "default.Table_17b597f08a264f22b57f12049ca81079" modify_external_catalogs: true
I20250112 20:51:30.432878 1186 log.cc:1198] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/cc971026f6b44db6af8215890ba71075
I20250112 20:51:30.433234 1186 ts_tablet_manager.cc:1939] T cc971026f6b44db6af8215890ba71075 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.433763 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f49b62d82ca74b20896a1eac33adf71a" } modify_external_catalogs: true
I20250112 20:51:30.434067 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4bd46a422093495ba8b5b03039874a32
I20250112 20:51:30.434294 1186 tablet_replica.cc:331] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:30.434681 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a64579137f154510b4d9b3432f9b3403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.434850 1186 raft_consensus.cc:2238] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.435006 1186 raft_consensus.cc:2267] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:30.435199 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a64579137f154510b4d9b3432f9b3403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.436326 1186 ts_tablet_manager.cc:1905] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.436411 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet cc971026f6b44db6af8215890ba71075 (table default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]) successfully deleted
I20250112 20:51:30.437218 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.427533 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:30.438043 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f49b62d82ca74b20896a1eac33adf71a" } modify_external_catalogs: true
I20250112 20:51:30.438141 911 raft_consensus.cc:3054] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.438242 1163 tablet_bootstrap.cc:492] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.438247 1178 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.438411 1163 ts_tablet_manager.cc:1397] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250112 20:51:30.438416 1178 ts_tablet_manager.cc:1397] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250112 20:51:30.439106 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.439158 1178 raft_consensus.cc:357] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.439345 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.439366 1178 raft_consensus.cc:383] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:30.439875 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4301a07fcb044fd2a6f3e5ef424b411f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4301a07fcb044fd2a6f3e5ef424b411f already in progress: creating tablet
I20250112 20:51:30.439972 1178 raft_consensus.cc:738] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.439976 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:30.440121 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4bd46a422093495ba8b5b03039874a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd46a422093495ba8b5b03039874a32 already in progress: creating tablet
I20250112 20:51:30.440201 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.439158 1163 raft_consensus.cc:357] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.440280 1178 consensus_queue.cc:260] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.440408 1163 raft_consensus.cc:383] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.440474 1163 raft_consensus.cc:738] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.440723 1163 consensus_queue.cc:260] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.440958 1178 ts_tablet_manager.cc:1428] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:30.441221 1178 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.441363 1163 ts_tablet_manager.cc:1428] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250112 20:51:30.441396 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4301a07fcb044fd2a6f3e5ef424b411f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4301a07fcb044fd2a6f3e5ef424b411f already in progress: creating tablet
W20250112 20:51:30.441635 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4bd46a422093495ba8b5b03039874a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd46a422093495ba8b5b03039874a32 already in progress: creating tablet
I20250112 20:51:30.441692 1163 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.441982 1430 raft_consensus.cc:491] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.442026 911 raft_consensus.cc:2463] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:30.442114 1430 raft_consensus.cc:513] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.442540 710 leader_election.cc:304] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.442745 1430 leader_election.cc:290] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.443073 1427 raft_consensus.cc:2798] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.443274 1427 raft_consensus.cc:695] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:30.443233 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0ba3d4af27c34f2e910508fc091eb463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.443361 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.443534 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:30.443745 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ba3d4af27c34f2e910508fc091eb463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.443773 773 raft_consensus.cc:2463] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.443743 1427 consensus_queue.cc:237] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.443992 1163 tablet_bootstrap.cc:654] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:30.444216 847 leader_election.cc:343] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.444660 847 leader_election.cc:304] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.444715 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_39bacb349abc4e21ab442b1e55db7a0e [id=9448c50d601f435e81de18e95f86efb0] alter complete (version 1)
I20250112 20:51:30.445132 1430 raft_consensus.cc:2798] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.445240 1430 raft_consensus.cc:491] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.445314 1430 raft_consensus.cc:3054] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.445446 451 master_service.cc:946] ReplaceTablet: received request to replace tablet d8330fa1e4654a1a9f167a1120a103fc from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.445925 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 35dcd1a120684446bbe19303fb8416ef from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.447996 1430 raft_consensus.cc:513] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.448513 1430 leader_election.cc:290] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.449179 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.449283 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.449456 773 raft_consensus.cc:3054] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.449805 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.449862 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 0871d4272e874a958174f96bfc0e1a95 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b7f79b3309e45afb9e18cad97ddfeb0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.450060 1184 tablet_replica.cc:331] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.450233 1184 raft_consensus.cc:2238] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.450383 1184 raft_consensus.cc:2267] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.450717 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d48b17628cf145b69e5fdde4131d9fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.451403 1184 ts_tablet_manager.cc:1905] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.452138 773 raft_consensus.cc:2463] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:30.452919 847 leader_election.cc:343] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.453140 847 leader_election.cc:304] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.453629 1430 raft_consensus.cc:2798] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.453897 1430 raft_consensus.cc:695] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.454185 1430 consensus_queue.cc:237] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.458062 1178 tablet_bootstrap.cc:654] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.458127 1184 ts_tablet_manager.cc:1918] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.458321 1184 log.cc:1198] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:30.458729 1184 ts_tablet_manager.cc:1939] T 953d2ec8a7564c0bbcf0166487dd1746 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.459781 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 953d2ec8a7564c0bbcf0166487dd1746 (table default.Table_40069cc52b264283ac3b145871a3b426 [id=8bd86656bb82483ea8cd34a80fa2b843]) successfully deleted
I20250112 20:51:30.460125 1184 tablet_replica.cc:331] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.460359 1184 raft_consensus.cc:2238] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.460592 1184 raft_consensus.cc:2267] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:30.461488 1184 ts_tablet_manager.cc:1905] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.463318 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.465540 1164 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.465744 1164 ts_tablet_manager.cc:1397] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.001s
I20250112 20:51:30.466573 1164 raft_consensus.cc:357] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.467024 1164 raft_consensus.cc:383] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.467159 1164 raft_consensus.cc:738] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.467429 1164 consensus_queue.cc:260] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.468047 1164 ts_tablet_manager.cc:1428] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.468429 1164 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.470142 1164 tablet_bootstrap.cc:654] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.473670 1163 tablet_bootstrap.cc:492] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.473923 1163 ts_tablet_manager.cc:1397] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.003s
I20250112 20:51:30.474946 1163 raft_consensus.cc:357] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.475279 1163 raft_consensus.cc:383] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.475440 1163 raft_consensus.cc:738] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.475800 1163 consensus_queue.cc:260] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.476670 1163 ts_tablet_manager.cc:1428] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:30.477062 1163 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.478428 1164 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.478638 1164 ts_tablet_manager.cc:1397] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250112 20:51:30.479423 1164 raft_consensus.cc:357] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.479712 1164 raft_consensus.cc:383] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.479844 1164 raft_consensus.cc:738] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.479838 1186 ts_tablet_manager.cc:1918] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.480000 1186 log.cc:1198] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:30.480226 1164 consensus_queue.cc:260] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.480933 1164 ts_tablet_manager.cc:1428] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:30.481277 1164 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.482671 1186 ts_tablet_manager.cc:1939] T b13d3f3e597f4876bce5852961f37ea0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:30.483989 1164 tablet_bootstrap.cc:654] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.483918 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b13d3f3e597f4876bce5852961f37ea0 (table default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]) successfully deleted
I20250112 20:51:30.481537 1178 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.484293 1178 ts_tablet_manager.cc:1397] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.000s
I20250112 20:51:30.485208 1178 raft_consensus.cc:357] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.485743 1178 raft_consensus.cc:383] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.485991 1163 tablet_bootstrap.cc:654] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:30.486435 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b310e4135c2b424fb006a38a60e41066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b310e4135c2b424fb006a38a60e41066 already in progress: creating tablet
I20250112 20:51:30.486721 1178 raft_consensus.cc:738] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.487203 1178 consensus_queue.cc:260] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:30.487960 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 31d3669355bc4b6bbc665edaa80c7a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31d3669355bc4b6bbc665edaa80c7a9e already in progress: creating tablet
I20250112 20:51:30.488543 1184 ts_tablet_manager.cc:1918] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.488749 1184 log.cc:1198] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/0871d4272e874a958174f96bfc0e1a95
W20250112 20:51:30.488837 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4bd46a422093495ba8b5b03039874a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd46a422093495ba8b5b03039874a32 already in progress: creating tablet
I20250112 20:51:30.489185 1184 ts_tablet_manager.cc:1939] T 0871d4272e874a958174f96bfc0e1a95 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.490252 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 0871d4272e874a958174f96bfc0e1a95 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
W20250112 20:51:30.490615 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4301a07fcb044fd2a6f3e5ef424b411f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4301a07fcb044fd2a6f3e5ef424b411f already in progress: creating tablet
I20250112 20:51:30.491029 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6caf53d8232c4928a6a597866d27c8dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.491209 1184 tablet_replica.cc:331] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:30.491245 1178 ts_tablet_manager.cc:1428] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:30.491384 1184 raft_consensus.cc:2238] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:30.491510 1178 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.491549 1184 raft_consensus.cc:2267] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:30.491643 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6caf53d8232c4928a6a597866d27c8dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.492033 1184 ts_tablet_manager.cc:1905] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:30.493171 1178 tablet_bootstrap.cc:654] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.497278 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_28f7cb2ca8314f618581e50f1e1bc16a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.497874 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28f7cb2ca8314f618581e50f1e1bc16a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.498277 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:30.499044 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1344f1b683b941899cf95663d76ffd18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.499215 1163 tablet_bootstrap.cc:492] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.499392 1163 ts_tablet_manager.cc:1397] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
W20250112 20:51:30.499544 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1344f1b683b941899cf95663d76ffd18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.499754 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:30.500120 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.500524 1163 raft_consensus.cc:357] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.500911 1163 raft_consensus.cc:383] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.501039 1163 raft_consensus.cc:738] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.501118 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.501365 1163 consensus_queue.cc:260] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.502612 1178 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.503010 1163 ts_tablet_manager.cc:1428] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
W20250112 20:51:30.502632 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 944b71897bcf4304bd9f7bb7adef34e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944b71897bcf4304bd9f7bb7adef34e5 already in progress: creating tablet
W20250112 20:51:30.503470 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 944b71897bcf4304bd9f7bb7adef34e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944b71897bcf4304bd9f7bb7adef34e5 already in progress: creating tablet
I20250112 20:51:30.504137 1178 ts_tablet_manager.cc:1397] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250112 20:51:30.505739 1163 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.508002 1164 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.508206 1164 ts_tablet_manager.cc:1397] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.007s
I20250112 20:51:30.509073 1164 raft_consensus.cc:357] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.509335 1164 raft_consensus.cc:383] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.509456 1164 raft_consensus.cc:738] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.509735 1164 consensus_queue.cc:260] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.511691 1164 ts_tablet_manager.cc:1428] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:30.512017 1164 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.514194 1164 tablet_bootstrap.cc:654] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.516822 1178 raft_consensus.cc:357] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.517076 1178 raft_consensus.cc:383] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.517167 1178 raft_consensus.cc:738] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.517256 1163 tablet_bootstrap.cc:654] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.517999 1178 consensus_queue.cc:260] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.518579 1178 ts_tablet_manager.cc:1428] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250112 20:51:30.518880 1178 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.520429 1178 tablet_bootstrap.cc:654] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.521744 1184 ts_tablet_manager.cc:1918] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:30.521958 1184 log.cc:1198] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d48b17628cf145b69e5fdde4131d9fd2
I20250112 20:51:30.522315 1164 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.522488 1184 ts_tablet_manager.cc:1939] T d48b17628cf145b69e5fdde4131d9fd2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:30.522503 1164 ts_tablet_manager.cc:1397] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250112 20:51:30.523205 1164 raft_consensus.cc:357] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.523550 1164 raft_consensus.cc:383] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.523674 1164 raft_consensus.cc:738] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.523926 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d48b17628cf145b69e5fdde4131d9fd2 (table default.Table_04f9f9491e5f49ec9660427758223e26 [id=d608132c961d49989093f7fd57baed12]) successfully deleted
I20250112 20:51:30.524012 1164 consensus_queue.cc:260] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.524705 1164 ts_tablet_manager.cc:1428] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:30.525065 1164 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:30.525327 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 944b71897bcf4304bd9f7bb7adef34e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944b71897bcf4304bd9f7bb7adef34e5 already in progress: creating tablet
I20250112 20:51:30.526744 1164 tablet_bootstrap.cc:654] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.527021 1163 tablet_bootstrap.cc:492] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.527185 1163 ts_tablet_manager.cc:1397] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250112 20:51:30.527909 1163 raft_consensus.cc:357] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.528136 1163 raft_consensus.cc:383] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.528244 1163 raft_consensus.cc:738] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.528489 1163 consensus_queue.cc:260] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.530257 1163 ts_tablet_manager.cc:1428] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:30.530555 1163 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.533468 1163 tablet_bootstrap.cc:654] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.536820 1164 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.537012 1164 ts_tablet_manager.cc:1397] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20250112 20:51:30.537768 1164 raft_consensus.cc:357] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.538015 1164 raft_consensus.cc:383] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.538129 1164 raft_consensus.cc:738] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.538388 1164 consensus_queue.cc:260] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.539837 1638 raft_consensus.cc:491] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.539968 1638 raft_consensus.cc:513] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.540628 1638 leader_election.cc:290] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.541424 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.542007 710 leader_election.cc:343] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.543540 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:30.544122 710 leader_election.cc:343] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.544232 710 leader_election.cc:304] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.544689 1427 raft_consensus.cc:2743] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.545310 1164 ts_tablet_manager.cc:1428] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250112 20:51:30.545632 1164 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.547837 1178 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.548022 1178 ts_tablet_manager.cc:1397] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250112 20:51:30.549324 1164 tablet_bootstrap.cc:654] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.550519 1178 raft_consensus.cc:357] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.550786 1178 raft_consensus.cc:383] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.550896 1178 raft_consensus.cc:738] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.558890 1164 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.559643 1164 ts_tablet_manager.cc:1397] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20250112 20:51:30.560396 1164 raft_consensus.cc:357] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.560726 1164 raft_consensus.cc:383] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.560879 1164 raft_consensus.cc:738] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.561221 1164 consensus_queue.cc:260] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.561453 1178 consensus_queue.cc:260] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.562052 1178 ts_tablet_manager.cc:1428] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 20:51:30.562422 1178 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.564426 1164 ts_tablet_manager.cc:1428] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:30.564795 1164 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.568320 1164 tablet_bootstrap.cc:654] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.571174 1163 tablet_bootstrap.cc:492] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.571404 1163 ts_tablet_manager.cc:1397] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.005s
I20250112 20:51:30.572116 1163 raft_consensus.cc:357] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.573287 1163 raft_consensus.cc:383] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.578649 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_62d64f27a517493eb41fe5974ca444ed" } new_table_name: "default.Table_a374fefd5800476d8e57e23cf0577a2a" modify_external_catalogs: true
I20250112 20:51:30.582321 1178 tablet_bootstrap.cc:654] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.584679 1163 raft_consensus.cc:738] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.586166 1627 raft_consensus.cc:491] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.586310 1627 raft_consensus.cc:513] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.587100 1627 leader_election.cc:290] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.587189 1163 consensus_queue.cc:260] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.587990 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.588420 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.588644 775 raft_consensus.cc:2463] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.589370 847 leader_election.cc:304] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:30.589687 847 leader_election.cc:343] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.590129 1163 ts_tablet_manager.cc:1428] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250112 20:51:30.590188 1627 raft_consensus.cc:2798] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.590306 1627 raft_consensus.cc:491] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.590422 1627 raft_consensus.cc:3054] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.591301 1163 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.593046 1163 tablet_bootstrap.cc:654] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.594882 1627 raft_consensus.cc:513] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.595927 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:30.596508 847 leader_election.cc:343] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.597568 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.597790 775 raft_consensus.cc:3054] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.608458 775 raft_consensus.cc:2463] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:30.610143 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]). Expected version 2 got 0
I20250112 20:51:30.610460 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:30.610659 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:30.610798 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
W20250112 20:51:30.612383 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81f0875e271d4aaf9dc5da6aa8113ed0 consensus state has no leader
I20250112 20:51:30.612856 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_98c777cc5e4149168e22f0635ec52fbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.613154 847 leader_election.cc:304] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:30.613401 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98c777cc5e4149168e22f0635ec52fbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.613443 1627 leader_election.cc:290] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.613767 1627 raft_consensus.cc:2798] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.614017 1627 raft_consensus.cc:695] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.614466 1163 tablet_bootstrap.cc:492] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.614612 1163 ts_tablet_manager.cc:1397] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.009s
I20250112 20:51:30.614840 1627 consensus_queue.cc:237] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.615615 1178 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.615780 1178 ts_tablet_manager.cc:1397] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250112 20:51:30.616247 1163 raft_consensus.cc:357] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.616441 1163 raft_consensus.cc:383] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.616528 1163 raft_consensus.cc:738] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.616535 1178 raft_consensus.cc:357] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.616729 1178 raft_consensus.cc:383] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.616751 1163 consensus_queue.cc:260] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.617030 1354 raft_consensus.cc:491] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.617146 1354 raft_consensus.cc:513] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.617782 1354 leader_election.cc:290] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.617723 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 30156185f6104eada87b936fc1829df9 (DEFAULT_TABLE table=default.Table_1b4f716257454c0bbc3266a5706d97e4 [id=ecdee533f4374b5b88f11e31a093fb79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.618486 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30156185f6104eada87b936fc1829df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.618572 753 tablet_service.cc:1467] Processing CreateTablet for tablet 30156185f6104eada87b936fc1829df9 (DEFAULT_TABLE table=default.Table_1b4f716257454c0bbc3266a5706d97e4 [id=ecdee533f4374b5b88f11e31a093fb79]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:30.618798 983 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 616b8aa8fd62486c978c275712064f77
I20250112 20:51:30.618985 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30156185f6104eada87b936fc1829df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.618963 1178 raft_consensus.cc:738] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.619231 1178 consensus_queue.cc:260] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.619822 1178 ts_tablet_manager.cc:1428] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:30.620116 1178 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.621779 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7fdddeb1e3534f599b5045b3a517b64c" } new_table_name: "default.Table_5860b3a08e52455f9708d82fe27361b9" modify_external_catalogs: true
W20250112 20:51:30.622081 983 leader_election.cc:343] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 616b8aa8fd62486c978c275712064f77
I20250112 20:51:30.622210 983 leader_election.cc:304] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.622557 1354 raft_consensus.cc:2743] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.626093 1638 raft_consensus.cc:491] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.626261 1638 raft_consensus.cc:513] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.626936 1638 leader_election.cc:290] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.627238 1163 ts_tablet_manager.cc:1428] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250112 20:51:30.627466 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.628027 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.628451 1163 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:30.628909 710 leader_election.cc:343] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.629153 710 leader_election.cc:343] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.629261 710 leader_election.cc:304] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.629607 1638 raft_consensus.cc:2743] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.630316 1163 tablet_bootstrap.cc:654] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.631738 1178 tablet_bootstrap.cc:654] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.633711 881 tablet_service.cc:1467] Processing CreateTablet for tablet 30156185f6104eada87b936fc1829df9 (DEFAULT_TABLE table=default.Table_1b4f716257454c0bbc3266a5706d97e4 [id=ecdee533f4374b5b88f11e31a093fb79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.634243 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30156185f6104eada87b936fc1829df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.637529 1638 raft_consensus.cc:491] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.637712 1638 raft_consensus.cc:513] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.638440 1638 leader_election.cc:290] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.644650 1178 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.644845 1178 ts_tablet_manager.cc:1397] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20250112 20:51:30.644971 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.645264 1164 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.645402 1164 ts_tablet_manager.cc:1397] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.020s
I20250112 20:51:30.645658 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.645846 911 raft_consensus.cc:2463] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:30.646346 1164 raft_consensus.cc:357] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.646862 710 leader_election.cc:304] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:30.646968 1164 raft_consensus.cc:383] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.647065 1164 raft_consensus.cc:738] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:30.647212 710 leader_election.cc:343] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.647290 1164 consensus_queue.cc:260] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.647652 1638 raft_consensus.cc:2798] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.647778 1638 raft_consensus.cc:491] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.647892 1638 raft_consensus.cc:3054] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.647984 1164 ts_tablet_manager.cc:1428] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:30.650779 1638 raft_consensus.cc:513] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.651387 1164 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.653359 1178 raft_consensus.cc:357] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.653558 1178 raft_consensus.cc:383] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.653627 1178 raft_consensus.cc:738] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.653856 1178 consensus_queue.cc:260] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.654342 1178 ts_tablet_manager.cc:1428] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 20:51:30.654595 1178 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.654347 1638 leader_election.cc:290] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.656148 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.656251 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:30.656435 911 raft_consensus.cc:3054] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:30.656842 710 leader_election.cc:343] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.659449 1164 tablet_bootstrap.cc:654] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.659618 911 raft_consensus.cc:2463] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:30.659942 1163 tablet_bootstrap.cc:492] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.660106 1163 ts_tablet_manager.cc:1397] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250112 20:51:30.660907 1163 raft_consensus.cc:357] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.661118 1163 raft_consensus.cc:383] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.661191 1163 raft_consensus.cc:738] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.661450 1163 consensus_queue.cc:260] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.661945 1163 ts_tablet_manager.cc:1428] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.663300 1163 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.665031 710 leader_election.cc:304] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.666286 1178 tablet_bootstrap.cc:654] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.668869 1164 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.669031 1164 ts_tablet_manager.cc:1397] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250112 20:51:30.669828 1164 raft_consensus.cc:357] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.670042 1164 raft_consensus.cc:383] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.670123 1164 raft_consensus.cc:738] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.670408 1164 consensus_queue.cc:260] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.672535 1427 raft_consensus.cc:2798] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.673251 1163 tablet_bootstrap.cc:654] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.685297 1164 ts_tablet_manager.cc:1428] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250112 20:51:30.686173 1164 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.688337 1164 tablet_bootstrap.cc:654] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.695305 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_40c44749c3934a83ab1b8ed501bf1c91" } new_table_name: "default.Table_df950a6830fa45b1a8f21feda53ea0ea" modify_external_catalogs: true
I20250112 20:51:30.697954 1163 tablet_bootstrap.cc:492] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.698211 1163 ts_tablet_manager.cc:1397] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.002s
I20250112 20:51:30.699134 1163 raft_consensus.cc:357] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.700771 1163 raft_consensus.cc:383] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.701208 1163 raft_consensus.cc:738] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.702047 1163 consensus_queue.cc:260] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.703501 1163 ts_tablet_manager.cc:1428] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:30.705317 1164 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.706490 1163 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.707298 1164 ts_tablet_manager.cc:1397] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20250112 20:51:30.709300 1163 tablet_bootstrap.cc:654] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.710338 1164 raft_consensus.cc:357] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.711462 1164 raft_consensus.cc:383] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.711756 1164 raft_consensus.cc:738] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.712321 1164 consensus_queue.cc:260] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:30.713469 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54ab0ae10f09469fa4f913e049002481 has no consensus state
I20250112 20:51:30.713774 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 304e1c8abe094bd99a99fd358b697f81 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.714363 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 4688dd6bbfc046b7bb140206cae9f104 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.719750 1627 raft_consensus.cc:491] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.721299 1164 ts_tablet_manager.cc:1428] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:30.721513 1627 raft_consensus.cc:513] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.721674 1164 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.722241 1627 leader_election.cc:290] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.722807 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.723296 847 leader_election.cc:343] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.725173 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.725679 775 raft_consensus.cc:2463] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.726111 847 leader_election.cc:304] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.726562 1627 raft_consensus.cc:2798] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.726655 1627 raft_consensus.cc:491] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.726765 1627 raft_consensus.cc:3054] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.727735 1163 tablet_bootstrap.cc:492] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.727957 1163 ts_tablet_manager.cc:1397] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250112 20:51:30.729115 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 54de54eabe5946058d1940b79f5fb832
I20250112 20:51:30.729559 1627 raft_consensus.cc:513] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.729741 1163 raft_consensus.cc:357] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.729880 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 953d2ec8a7564c0bbcf0166487dd1746 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.729974 1163 raft_consensus.cc:383] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.730046 1163 raft_consensus.cc:738] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.731036 1627 leader_election.cc:290] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.731097 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.731320 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.731464 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:30.732040 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 54de54eabe5946058d1940b79f5fb832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54de54eabe5946058d1940b79f5fb832 already in progress: creating tablet
W20250112 20:51:30.732334 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 54de54eabe5946058d1940b79f5fb832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54de54eabe5946058d1940b79f5fb832 already in progress: creating tablet
I20250112 20:51:30.732416 1163 consensus_queue.cc:260] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:30.732682 847 leader_election.cc:343] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.732964 1163 ts_tablet_manager.cc:1428] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:30.733875 1178 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.734091 1178 ts_tablet_manager.cc:1397] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.001s
I20250112 20:51:30.734128 1163 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.734899 1178 raft_consensus.cc:357] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.735119 1178 raft_consensus.cc:383] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.735188 1178 raft_consensus.cc:738] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.735428 1178 consensus_queue.cc:260] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.736094 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a5ce6ae642344a2c950ae993e6d90311" } new_table_name: "default.Table_15828ef1bbbc41068bb6b6f2fbb18f94" modify_external_catalogs: true
I20250112 20:51:30.740751 1178 ts_tablet_manager.cc:1428] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:30.741884 1163 tablet_bootstrap.cc:654] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.743045 1178 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.744663 1178 tablet_bootstrap.cc:654] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.745934 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.746165 775 raft_consensus.cc:3054] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.750530 775 raft_consensus.cc:2463] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:30.749002 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.751003 847 leader_election.cc:304] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.751399 1627 raft_consensus.cc:2798] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.751539 1627 raft_consensus.cc:695] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.751596 1164 tablet_bootstrap.cc:654] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.751776 1163 tablet_bootstrap.cc:492] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.751775 1627 consensus_queue.cc:237] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.751932 1163 ts_tablet_manager.cc:1397] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20250112 20:51:30.752715 1163 raft_consensus.cc:357] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:30.752853 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 54de54eabe5946058d1940b79f5fb832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54de54eabe5946058d1940b79f5fb832 already in progress: creating tablet
I20250112 20:51:30.752916 1163 raft_consensus.cc:383] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.752988 1163 raft_consensus.cc:738] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.755040 1163 consensus_queue.cc:260] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.757253 1427 raft_consensus.cc:695] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:30.760736 1427 consensus_queue.cc:237] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.762539 1163 ts_tablet_manager.cc:1428] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 20:51:30.763753 1163 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.769848 1163 tablet_bootstrap.cc:654] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.772711 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0ba3d4af27c34f2e910508fc091eb463" } modify_external_catalogs: true
I20250112 20:51:30.773947 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0ba3d4af27c34f2e910508fc091eb463" } modify_external_catalogs: true
I20250112 20:51:30.772886 1613 raft_consensus.cc:491] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.776118 1613 raft_consensus.cc:513] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.776947 1613 leader_election.cc:290] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.777904 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.778033 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca061b82a894e91b9508613ccd8ecd7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.778297 775 raft_consensus.cc:2388] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:30.778730 983 leader_election.cc:304] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.779115 1613 raft_consensus.cc:3054] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.782624 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:30.783176 451 catalog_manager.cc:6596] ReplaceTablet: tablet 35dcd1a120684446bbe19303fb8416ef deleted and replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6
I20250112 20:51:30.783689 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.784409 1613 raft_consensus.cc:2743] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:30.784947 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 35dcd1a120684446bbe19303fb8416ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35dcd1a120684446bbe19303fb8416ef already in progress: creating tablet
I20250112 20:51:30.785250 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:30.785738 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:30.785985 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 35dcd1a120684446bbe19303fb8416ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35dcd1a120684446bbe19303fb8416ef already in progress: creating tablet
I20250112 20:51:30.786375 1178 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.786537 1178 ts_tablet_manager.cc:1397] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.009s
W20250112 20:51:30.786684 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 35dcd1a120684446bbe19303fb8416ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35dcd1a120684446bbe19303fb8416ef already in progress: creating tablet
I20250112 20:51:30.787377 1178 raft_consensus.cc:357] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.787593 1178 raft_consensus.cc:383] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.787731 1178 raft_consensus.cc:738] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.787995 1178 consensus_queue.cc:260] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.788453 1663 raft_consensus.cc:491] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.788678 1663 raft_consensus.cc:513] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.790477 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:30.790990 710 leader_election.cc:343] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.791633 1663 leader_election.cc:290] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.791954 1178 ts_tablet_manager.cc:1428] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:51:30.792065 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.792244 1178 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:30.792884 710 leader_election.cc:343] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.793009 710 leader_election.cc:304] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.798069 1178 tablet_bootstrap.cc:654] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.799149 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_23317c504a8b4a7392676291f5385b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.799805 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23317c504a8b4a7392676291f5385b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.802686 1663 raft_consensus.cc:2743] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.804857 1163 tablet_bootstrap.cc:492] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.805150 1163 ts_tablet_manager.cc:1397] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.004s
I20250112 20:51:30.805577 1178 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:30.805763 1178 ts_tablet_manager.cc:1397] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250112 20:51:30.806442 1163 raft_consensus.cc:357] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.806499 1178 raft_consensus.cc:357] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.806737 1178 raft_consensus.cc:383] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.806819 1178 raft_consensus.cc:738] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.809021 1163 raft_consensus.cc:383] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.809448 1163 raft_consensus.cc:738] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.809772 1163 consensus_queue.cc:260] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.812770 1613 raft_consensus.cc:491] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.813464 1613 raft_consensus.cc:513] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.815940 1613 leader_election.cc:290] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.816996 1164 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.816972 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.817137 1164 ts_tablet_manager.cc:1397] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.002s
I20250112 20:51:30.817988 1164 raft_consensus.cc:357] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.818271 1164 raft_consensus.cc:383] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.818384 1164 raft_consensus.cc:738] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.818701 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b414be57990432bb786daf5faf0cc2e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.818720 1164 consensus_queue.cc:260] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.818945 775 raft_consensus.cc:2388] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:30.819294 1164 ts_tablet_manager.cc:1428] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:30.819411 983 leader_election.cc:304] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.819584 1164 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.819927 1613 raft_consensus.cc:3054] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.820006 1178 consensus_queue.cc:260] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.821305 1164 tablet_bootstrap.cc:654] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.822634 1613 raft_consensus.cc:2743] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.823736 1163 ts_tablet_manager.cc:1428] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20250112 20:51:30.824163 1178 ts_tablet_manager.cc:1428] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 20:51:30.824476 1178 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:30.825881 1163 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.826241 1613 raft_consensus.cc:491] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.826509 1613 raft_consensus.cc:513] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:30.827914 983 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: b13d3f3e597f4876bce5852961f37ea0
W20250112 20:51:30.829056 983 leader_election.cc:343] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:30.829170 983 leader_election.cc:304] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.829480 1354 raft_consensus.cc:2743] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.830348 1613 leader_election.cc:290] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:30.832566 1163 tablet_bootstrap.cc:654] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:30.836166 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 81f0875e271d4aaf9dc5da6aa8113ed0 consensus state has no leader
I20250112 20:51:30.845479 1178 tablet_bootstrap.cc:654] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:30.845894 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f720bd9710d2476d9632699a2732cb05 consensus state has no leader
I20250112 20:51:30.846426 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_65e8cf7dc1134afaae922506cf15579b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.847095 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65e8cf7dc1134afaae922506cf15579b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:30.851154 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a218ef5825424b1b84c55b38b9b9023d consensus state has no leader
I20250112 20:51:30.852411 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_941f7aeb51a54bf38161efd28eb35340" } modify_external_catalogs: true
I20250112 20:51:30.852620 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_941f7aeb51a54bf38161efd28eb35340" } modify_external_catalogs: true
I20250112 20:51:30.860690 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:30.861318 446 catalog_manager.cc:6596] ReplaceTablet: tablet 4688dd6bbfc046b7bb140206cae9f104 deleted and replaced by tablet d477fb5e5ad5447bb182bc9998561a43
I20250112 20:51:30.865442 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:30.865723 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:30.866161 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4688dd6bbfc046b7bb140206cae9f104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4688dd6bbfc046b7bb140206cae9f104 already in progress: creating tablet
I20250112 20:51:30.866561 1427 raft_consensus.cc:491] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.866734 1427 raft_consensus.cc:513] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.867448 1427 leader_election.cc:290] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.868165 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:30.868944 710 leader_election.cc:343] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.869452 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4688dd6bbfc046b7bb140206cae9f104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4688dd6bbfc046b7bb140206cae9f104 already in progress: creating tablet
I20250112 20:51:30.869647 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:30.870045 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:30.870697 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4688dd6bbfc046b7bb140206cae9f104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4688dd6bbfc046b7bb140206cae9f104 already in progress: creating tablet
W20250112 20:51:30.871119 710 leader_election.cc:343] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.871243 710 leader_election.cc:304] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.871693 1427 raft_consensus.cc:2743] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.874219 1613 raft_consensus.cc:491] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.874426 1613 raft_consensus.cc:513] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.875025 1613 leader_election.cc:290] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:30.876327 983 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 29b70ffb3e3341d398d36ff76993e9ac
W20250112 20:51:30.876693 983 leader_election.cc:343] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:30.876838 983 leader_election.cc:304] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:30.877298 1613 raft_consensus.cc:2743] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.881660 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a18054e19f524ebd88965aa3727a35f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.882462 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a18054e19f524ebd88965aa3727a35f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:30.891166 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e72fda1743fc4f96b29add6e36e884de consensus state has no leader
I20250112 20:51:30.891516 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 81f0875e271d4aaf9dc5da6aa8113ed0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:30.892835 1427 raft_consensus.cc:491] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.893023 1427 raft_consensus.cc:513] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.893682 1427 leader_election.cc:290] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.894909 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.895552 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.896644 710 leader_election.cc:343] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.896926 710 leader_election.cc:343] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.897033 710 leader_election.cc:304] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.897507 1427 raft_consensus.cc:2743] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.908965 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet dd8dec7242fb4b44a06b9a2f64d6b6ca
I20250112 20:51:30.929615 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu pid 563
I20250112 20:51:30.929724 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.930315 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.930806 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:30.937705 1427 raft_consensus.cc:491] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.937901 1427 raft_consensus.cc:513] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.938608 1427 leader_election.cc:290] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.940855 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.941076 911 raft_consensus.cc:2463] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:30.941409 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.942162 710 leader_election.cc:343] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.942366 710 leader_election.cc:304] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.942994 1427 raft_consensus.cc:2798] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.943094 1427 raft_consensus.cc:491] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.943179 1427 raft_consensus.cc:3054] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.946375 1427 raft_consensus.cc:513] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.948396 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.949007 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.949164 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:30.949362 911 raft_consensus.cc:3054] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.950665 1627 raft_consensus.cc:491] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.950827 1627 raft_consensus.cc:513] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.951532 1627 leader_election.cc:290] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.952158 911 raft_consensus.cc:2463] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:30.952258 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.952858 847 leader_election.cc:343] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.953150 1427 leader_election.cc:290] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.953327 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8db2ac83bd80420da5eed5abfe3c3fda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.953975 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db2ac83bd80420da5eed5abfe3c3fda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.958132 1164 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:30.958287 1163 tablet_bootstrap.cc:492] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.958341 1164 ts_tablet_manager.cc:1397] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.000s
I20250112 20:51:30.953243 710 leader_election.cc:304] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:30.959407 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:30.959633 775 raft_consensus.cc:2463] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:30.960685 847 leader_election.cc:304] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.961216 1627 raft_consensus.cc:2798] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:30.961335 1627 raft_consensus.cc:491] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:30.961408 1627 raft_consensus.cc:3054] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.964540 1627 raft_consensus.cc:513] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.960176 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.965354 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 717aaf9875ef4abe92ec555f71ee85d6
I20250112 20:51:30.965416 1163 ts_tablet_manager.cc:1397] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.140s user 0.022s sys 0.041s
W20250112 20:51:30.965559 710 leader_election.cc:343] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.965672 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_70dd4d25b3854472992074db0c6fbeb3" } modify_external_catalogs: true
I20250112 20:51:30.965863 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_70dd4d25b3854472992074db0c6fbeb3" } modify_external_catalogs: true
I20250112 20:51:30.966382 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:30.966501 1163 raft_consensus.cc:357] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.966707 1163 raft_consensus.cc:383] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.966639 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c37a19db737147b3a05d4cd80380f59e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.966789 1163 raft_consensus.cc:738] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:30.966878 847 leader_election.cc:343] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.966998 1163 consensus_queue.cc:260] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:30.967133 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c37a19db737147b3a05d4cd80380f59e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.967561 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:30.967811 775 raft_consensus.cc:3054] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:30.968910 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a476af27be9d446dab54b21d797ca38f" } new_table_name: "default.Table_800fbaf988064a7e8810c2617a6c8863" modify_external_catalogs: true
I20250112 20:51:30.970181 1163 ts_tablet_manager.cc:1428] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:30.970486 1163 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.970510 775 raft_consensus.cc:2463] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:30.973487 847 leader_election.cc:304] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:30.973789 1430 raft_consensus.cc:2798] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.973945 1430 raft_consensus.cc:695] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:30.974188 1430 consensus_queue.cc:237] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.978308 1163 tablet_bootstrap.cc:654] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.981099 1627 leader_election.cc:290] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:30.981508 1427 raft_consensus.cc:2798] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:30.981856 1427 raft_consensus.cc:695] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:30.982156 1427 consensus_queue.cc:237] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:30.982474 1638 raft_consensus.cc:491] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:30.982585 1164 raft_consensus.cc:357] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.982666 1638 raft_consensus.cc:513] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.983029 1163 tablet_bootstrap.cc:492] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:30.983211 1163 ts_tablet_manager.cc:1397] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20250112 20:51:30.983399 1638 leader_election.cc:290] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:30.983737 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:30.983865 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:30.984447 710 leader_election.cc:343] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:30.984640 710 leader_election.cc:343] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:30.984748 710 leader_election.cc:304] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:30.986240 1638 raft_consensus.cc:2743] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:30.987490 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6d7de66040254347bff24c5702f6241c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:30.987543 1163 raft_consensus.cc:357] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.987788 1163 raft_consensus.cc:383] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.987855 1163 raft_consensus.cc:738] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:30.988097 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d7de66040254347bff24c5702f6241c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.988086 1163 consensus_queue.cc:260] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:30.982826 1164 raft_consensus.cc:383] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.988682 1164 raft_consensus.cc:738] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.989025 1164 consensus_queue.cc:260] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:30.989797 1163 ts_tablet_manager.cc:1428] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250112 20:51:30.990150 1163 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:30.991874 1164 ts_tablet_manager.cc:1428] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20250112 20:51:30.992166 1164 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:30.992285 1163 tablet_bootstrap.cc:654] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.992811 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_39c63dc2cbe1434f83e4872d1fba7757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:30.993444 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39c63dc2cbe1434f83e4872d1fba7757 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.995616 1164 tablet_bootstrap.cc:654] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.000164 1164 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.000341 1164 ts_tablet_manager.cc:1397] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250112 20:51:31.001161 1164 raft_consensus.cc:357] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.001389 1164 raft_consensus.cc:383] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.001482 1164 raft_consensus.cc:738] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.001695 1164 consensus_queue.cc:260] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.002674 1164 ts_tablet_manager.cc:1428] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:31.002972 1164 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:31.003032 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55d75f9b410f4790ac926646a3cf0fd0 consensus state has no leader
I20250112 20:51:31.003409 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2dc9d59ea355480789bf06df8c358e57" } new_table_name: "default.Table_b2ca205a6c6f470c896d5194717dbf72" modify_external_catalogs: true
I20250112 20:51:31.003873 1163 tablet_bootstrap.cc:492] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.004011 1163 ts_tablet_manager.cc:1397] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20250112 20:51:31.004951 1163 raft_consensus.cc:357] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.005169 1163 raft_consensus.cc:383] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.005259 1163 raft_consensus.cc:738] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.006203 1164 tablet_bootstrap.cc:654] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.006685 1627 raft_consensus.cc:491] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.006860 1627 raft_consensus.cc:513] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.007527 1627 leader_election.cc:290] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.007570 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_144f2be878e14cb2901b6993ebee9191" } modify_external_catalogs: true
I20250112 20:51:31.007789 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_144f2be878e14cb2901b6993ebee9191" } modify_external_catalogs: true
I20250112 20:51:31.005491 1163 consensus_queue.cc:260] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.014832 1163 ts_tablet_manager.cc:1428] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 20:51:31.015125 1163 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.016757 1427 raft_consensus.cc:491] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.016817 1163 tablet_bootstrap.cc:654] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.007982 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.016913 1427 raft_consensus.cc:513] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:31.017345 847 leader_election.cc:343] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.017511 1427 leader_election.cc:290] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.023890 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304e1c8abe094bd99a99fd358b697f81" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.024187 911 raft_consensus.cc:2463] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.025149 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304e1c8abe094bd99a99fd358b697f81" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.025416 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.025657 775 raft_consensus.cc:2463] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.026150 847 leader_election.cc:304] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:31.026172 710 leader_election.cc:343] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.026371 710 leader_election.cc:304] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.026472 1163 tablet_bootstrap.cc:492] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.026542 1627 raft_consensus.cc:2798] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.026618 1163 ts_tablet_manager.cc:1397] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20250112 20:51:31.026616 1427 raft_consensus.cc:2798] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.026643 1627 raft_consensus.cc:491] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.026717 1627 raft_consensus.cc:3054] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.026705 1427 raft_consensus.cc:491] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.026813 1427 raft_consensus.cc:3054] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.027609 1163 raft_consensus.cc:357] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.027842 1163 raft_consensus.cc:383] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.027909 1163 raft_consensus.cc:738] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.028134 1163 consensus_queue.cc:260] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.028713 1163 ts_tablet_manager.cc:1428] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:31.029034 1163 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.029474 1427 raft_consensus.cc:513] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.030081 1427 leader_election.cc:290] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.030066 1627 raft_consensus.cc:513] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.030645 1163 tablet_bootstrap.cc:654] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.031061 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.031080 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304e1c8abe094bd99a99fd358b697f81" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.031265 911 raft_consensus.cc:3054] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.031560 847 leader_election.cc:343] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.033190 1627 leader_election.cc:290] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.033586 1163 tablet_bootstrap.cc:492] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.033756 1163 ts_tablet_manager.cc:1397] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250112 20:51:31.033888 1627 raft_consensus.cc:491] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.034026 1627 raft_consensus.cc:513] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.034466 1163 raft_consensus.cc:357] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:31.034705 1163 raft_consensus.cc:383] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.034804 1163 raft_consensus.cc:738] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.034828 911 raft_consensus.cc:2463] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:31.034994 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.035499 1627 leader_election.cc:290] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.035069 1163 consensus_queue.cc:260] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:31.035197 1044 raft_consensus.cc:2463] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.037516 1163 ts_tablet_manager.cc:1428] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:31.037536 847 leader_election.cc:304] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.037791 1163 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.039122 1627 raft_consensus.cc:2798] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.039224 1627 raft_consensus.cc:491] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.039292 1627 raft_consensus.cc:3054] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.039693 448 master_service.cc:946] ReplaceTablet: received request to replace tablet d770e0399e80491f8de9b08d1eb2f139 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:31.040555 710 leader_election.cc:304] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.041023 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304e1c8abe094bd99a99fd358b697f81" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.041280 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.041484 775 raft_consensus.cc:3054] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.041574 1163 tablet_bootstrap.cc:654] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.043262 1627 raft_consensus.cc:513] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.044211 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.044411 1044 raft_consensus.cc:3054] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.045224 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.045446 773 raft_consensus.cc:2463] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:31.045899 710 leader_election.cc:343] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.046289 1427 raft_consensus.cc:2798] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.046429 1427 raft_consensus.cc:695] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.046777 1427 consensus_queue.cc:237] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.048434 775 raft_consensus.cc:2463] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.051777 1163 tablet_bootstrap.cc:492] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.052031 1163 ts_tablet_manager.cc:1397] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20250112 20:51:31.052829 1163 raft_consensus.cc:357] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.053077 1163 raft_consensus.cc:383] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.053158 1163 raft_consensus.cc:738] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.053205 1044 raft_consensus.cc:2463] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.053421 1163 consensus_queue.cc:260] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.053608 847 leader_election.cc:304] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.053925 1430 raft_consensus.cc:2798] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.054046 1430 raft_consensus.cc:695] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.054078 1163 ts_tablet_manager.cc:1428] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.054356 1163 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.054277 1430 consensus_queue.cc:237] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.056016 1163 tablet_bootstrap.cc:654] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:31.056392 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24c8dc85ac9240598d6e413deac19abe consensus state has no leader
I20250112 20:51:31.056677 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 304e1c8abe094bd99a99fd358b697f81 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:31.057790 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.058367 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.064817 1627 leader_election.cc:290] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.067474 1163 tablet_bootstrap.cc:492] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.067675 1163 ts_tablet_manager.cc:1397] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20250112 20:51:31.067907 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f4cae067df1a40098f81cfbd17224c75 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:31.068506 1163 raft_consensus.cc:357] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.068699 1163 raft_consensus.cc:383] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.068832 1163 raft_consensus.cc:738] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.069096 1163 consensus_queue.cc:260] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.069306 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_49d04bb236144efe8f33e836bc8d2483" } new_table_name: "default.Table_0a07af8986ee4235a0240a69d577e2c4" modify_external_catalogs: true
I20250112 20:51:31.070653 847 leader_election.cc:304] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.071126 1587 raft_consensus.cc:491] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.071270 1587 raft_consensus.cc:513] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.071806 1587 leader_election.cc:290] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.071971 1587 raft_consensus.cc:2798] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.072090 1587 raft_consensus.cc:695] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.072314 1587 consensus_queue.cc:237] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.072989 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.073194 1044 raft_consensus.cc:2463] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.073451 1163 ts_tablet_manager.cc:1428] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250112 20:51:31.073693 1163 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.075417 1163 tablet_bootstrap.cc:654] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.075750 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f39195945024f4fba9942faec70bc31" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.075969 775 raft_consensus.cc:3054] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.076465 847 leader_election.cc:304] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.078600 775 raft_consensus.cc:2463] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.079543 1627 raft_consensus.cc:2798] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.079659 1627 raft_consensus.cc:491] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.079663 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.079778 1627 raft_consensus.cc:3054] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.079887 773 raft_consensus.cc:2463] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.086108 1044 raft_consensus.cc:3054] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.086634 1627 raft_consensus.cc:513] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.087705 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.088121 1048 raft_consensus.cc:3054] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.089251 1627 leader_election.cc:290] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.091010 1048 raft_consensus.cc:2463] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.091470 847 leader_election.cc:304] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.093008 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c697bba271f4455956b9a2511155b79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.093205 773 raft_consensus.cc:3054] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.093778 1430 raft_consensus.cc:2798] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.093924 1430 raft_consensus.cc:695] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.094466 1430 consensus_queue.cc:237] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.095643 773 raft_consensus.cc:2463] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.096477 1163 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:31.096653 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d477fb5e5ad5447bb182bc9998561a43 has no consensus state
I20250112 20:51:31.096688 1163 ts_tablet_manager.cc:1397] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250112 20:51:31.097643 1163 raft_consensus.cc:357] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.097913 1163 raft_consensus.cc:383] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.097867 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f0c0c02b146a40f38d0098e259fe4cb6" } new_table_name: "default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f" modify_external_catalogs: true
I20250112 20:51:31.098024 1163 raft_consensus.cc:738] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.098310 1163 consensus_queue.cc:260] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.093906 1694 consensus_queue.cc:1035] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250112 20:51:31.099716 1427 raft_consensus.cc:491] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.099948 1427 raft_consensus.cc:513] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.100790 1427 leader_election.cc:290] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.101498 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.101887 911 raft_consensus.cc:2463] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.102617 710 leader_election.cc:304] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.102612 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:31.103207 710 leader_election.cc:343] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.104533 1427 raft_consensus.cc:2798] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.104705 1427 raft_consensus.cc:491] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.104887 1427 raft_consensus.cc:3054] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.106257 1163 ts_tablet_manager.cc:1428] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250112 20:51:31.106570 1163 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.108144 1163 tablet_bootstrap.cc:654] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.113940 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:31.113988 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 64887b3578284cb286ee122ab8c626c4
I20250112 20:51:31.114482 455 catalog_manager.cc:6596] ReplaceTablet: tablet 81f0875e271d4aaf9dc5da6aa8113ed0 deleted and replaced by tablet 9856e6b625d04f17b0869cee08f69142
I20250112 20:51:31.114794 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_395d5be237c54baea392437bef2cbadf" } new_table_name: "default.Table_79e838bb6ced422285e3bb8e26367f4f" modify_external_catalogs: true
I20250112 20:51:31.114920 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.115612 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:31.116485 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f0875e271d4aaf9dc5da6aa8113ed0 already in progress: creating tablet
W20250112 20:51:31.116742 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f0875e271d4aaf9dc5da6aa8113ed0 already in progress: creating tablet
I20250112 20:51:31.117318 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.117524 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.118680 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6caf53d8232c4928a6a597866d27c8dd" } modify_external_catalogs: true
I20250112 20:51:31.119494 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.120177 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:31.119180 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 64887b3578284cb286ee122ab8c626c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64887b3578284cb286ee122ab8c626c4 already in progress: creating tablet
W20250112 20:51:31.120986 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f0875e271d4aaf9dc5da6aa8113ed0 already in progress: creating tablet
W20250112 20:51:31.121183 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 64887b3578284cb286ee122ab8c626c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64887b3578284cb286ee122ab8c626c4 already in progress: creating tablet
W20250112 20:51:31.121326 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 64887b3578284cb286ee122ab8c626c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64887b3578284cb286ee122ab8c626c4 already in progress: creating tablet
I20250112 20:51:31.121543 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6caf53d8232c4928a6a597866d27c8dd" } modify_external_catalogs: true
I20250112 20:51:31.123667 1663 raft_consensus.cc:491] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.123849 1663 raft_consensus.cc:513] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.124498 1663 leader_election.cc:290] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.124431 1430 consensus_queue.cc:1035] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.125133 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:31.125936 710 leader_election.cc:343] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.125944 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.126432 1694 raft_consensus.cc:491] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.126652 1694 raft_consensus.cc:513] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.127471 1694 leader_election.cc:290] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.128556 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.128829 1045 raft_consensus.cc:2463] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.129604 847 leader_election.cc:304] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.130026 1694 raft_consensus.cc:2798] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.130216 1694 raft_consensus.cc:491] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.130376 1694 raft_consensus.cc:3054] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.133441 1178 tablet_bootstrap.cc:492] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.133584 1178 ts_tablet_manager.cc:1397] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.309s user 0.036s sys 0.031s
W20250112 20:51:31.133791 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64e1c32cab2c4b438d3ff4a7e674feeb consensus state has no leader
I20250112 20:51:31.134588 1178 raft_consensus.cc:357] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.134835 1178 raft_consensus.cc:383] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.134950 1178 raft_consensus.cc:738] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.135246 1178 consensus_queue.cc:260] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.135965 1178 ts_tablet_manager.cc:1428] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:31.136303 1178 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.137753 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:31.138001 453 catalog_manager.cc:5526] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.138417 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
I20250112 20:51:31.138641 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:31.138824 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:31.138957 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
I20250112 20:51:31.139215 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:31.139411 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_9c771feed27441ec8340f7279a4a06c1 [id=2ee1f97fef0f4e2b83932132d0936504] alter complete (version 2)
I20250112 20:51:31.139911 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:31.140246 1178 tablet_bootstrap.cc:654] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.141165 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.141688 1186 tablet_replica.cc:331] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:31.141908 1186 raft_consensus.cc:2238] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:31.142194 1186 pending_rounds.cc:62] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:31.142262 1186 pending_rounds.cc:69] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585009583968256 op_type: NO_OP noop_request { }
I20250112 20:51:31.142397 1186 raft_consensus.cc:2883] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:31.142503 1186 raft_consensus.cc:2267] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.142526 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.142784 1185 tablet_replica.cc:331] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.142961 1185 raft_consensus.cc:2238] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.143123 1185 raft_consensus.cc:2267] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.143682 1185 ts_tablet_manager.cc:1905] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.143805 1186 ts_tablet_manager.cc:1905] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.145699 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.145792 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f1c76db0d64624bd43f0e538f0649d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.145905 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 953d2ec8a7564c0bbcf0166487dd1746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.145787 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a692cc5fb6f4afc854c4821274edfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.146090 1184 tablet_replica.cc:331] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:31.146109 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 616b8aa8fd62486c978c275712064f77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.152230 1694 raft_consensus.cc:513] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.153237 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aa4af444eb4c4982a8ea14d673f83647"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.153839 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4af444eb4c4982a8ea14d673f83647 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.153849 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.154072 1045 raft_consensus.cc:3054] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.154229 1185 ts_tablet_manager.cc:1918] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.154343 1185 log.cc:1198] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:31.154762 1185 ts_tablet_manager.cc:1939] T 2afa5b0717df4f83a2b44efac12fc06b P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:31.153323 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.155411 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.155858 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
I20250112 20:51:31.156905 1045 raft_consensus.cc:2463] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.157617 847 leader_election.cc:304] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.158012 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.158234 773 raft_consensus.cc:2463] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.158295 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:31.158567 710 leader_election.cc:343] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.158849 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f34aa32d6474561988b2cf3d0e8fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.159046 710 leader_election.cc:304] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.160169 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.161062 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f34aa32d6474561988b2cf3d0e8fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.161168 1430 raft_consensus.cc:2798] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.161386 1430 raft_consensus.cc:695] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.161760 1430 consensus_queue.cc:237] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.166394 1694 leader_election.cc:290] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.169198 1663 raft_consensus.cc:2743] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.171239 1613 raft_consensus.cc:491] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.172796 1613 raft_consensus.cc:513] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.173496 1613 leader_election.cc:290] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.174464 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ab141f89a5bc48c1b0e8d83f0975d74f" } modify_external_catalogs: true
I20250112 20:51:31.174576 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.174726 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ab141f89a5bc48c1b0e8d83f0975d74f" } modify_external_catalogs: true
I20250112 20:51:31.175274 911 raft_consensus.cc:2388] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:31.177408 1613 raft_consensus.cc:491] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.177577 1613 raft_consensus.cc:513] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.178381 1613 leader_election.cc:290] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:31.179548 983 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: cc971026f6b44db6af8215890ba71075
I20250112 20:51:31.185520 1186 ts_tablet_manager.cc:1918] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:31.188320 1186 log.cc:1198] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:31.188412 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.188875 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f34aa32d6474561988b2cf3d0e8fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.191267 1186 ts_tablet_manager.cc:1939] T 2afa5b0717df4f83a2b44efac12fc06b P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:31.194133 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
I20250112 20:51:31.198730 1178 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.198967 1178 ts_tablet_manager.cc:1397] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.000s
I20250112 20:51:31.200248 1178 raft_consensus.cc:357] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.200592 1178 raft_consensus.cc:383] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.200701 1178 raft_consensus.cc:738] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.201071 1178 consensus_queue.cc:260] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.201107 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decdec0d34344ec4b422643b0d4e7659" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.201323 773 raft_consensus.cc:3054] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.205983 773 raft_consensus.cc:2463] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.208511 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc920e92de545fc961f82e7d0de28cd (DEFAULT_TABLE table=default.Table_211ca26c30e94a03aaf75698f6765585 [id=9dda94626ac842fabfa576cfb417dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.209022 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc920e92de545fc961f82e7d0de28cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.210980 1663 raft_consensus.cc:491] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.211230 1663 raft_consensus.cc:513] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.212055 1663 leader_election.cc:290] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.212714 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef89229192934339954566e366b6891c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.215924 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:31.216640 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.217240 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.220461 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.204075 1701 raft_consensus.cc:491] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.223538 1701 raft_consensus.cc:513] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.224577 1701 leader_election.cc:290] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.221029 1430 raft_consensus.cc:491] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.226516 1430 raft_consensus.cc:513] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.227761 1613 raft_consensus.cc:491] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.227921 1613 raft_consensus.cc:513] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.227828 1430 leader_election.cc:290] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.228482 1613 leader_election.cc:290] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.229683 882 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:31.229821 983 leader_election.cc:343] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:31.230171 753 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.230475 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.230753 1045 raft_consensus.cc:2463] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.230741 881 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc920e92de545fc961f82e7d0de28cd (DEFAULT_TABLE table=default.Table_211ca26c30e94a03aaf75698f6765585 [id=9dda94626ac842fabfa576cfb417dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.231129 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc920e92de545fc961f82e7d0de28cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.231221 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.231482 1049 raft_consensus.cc:2463] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.231592 1613 raft_consensus.cc:491] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.231765 1613 raft_consensus.cc:513] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.232364 1613 leader_election.cc:290] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.233011 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.232990 847 leader_election.cc:304] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.237174 847 leader_election.cc:304] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.237486 1700 raft_consensus.cc:2798] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 20:51:31.237953 449 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 33a90f801d854d4d8f7f09112e2d2929 consensus state has no leader
I20250112 20:51:31.238281 1700 raft_consensus.cc:491] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.238469 1700 raft_consensus.cc:3054] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.238598 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 25) took 994 ms (client timeout 1000 ms). Trace:
I20250112 20:51:31.129179 1427 raft_consensus.cc:513] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:31.238706 449 rpcz_store.cc:269] 0112 20:51:30.244250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:30.257752 (+ 13502us) service_pool.cc:225] Handling call
0112 20:51:30.312635 (+ 54883us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14f96b8 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:31.238519 (+925884us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":10,"spinlock_wait_cycles":124279040}
I20250112 20:51:31.239905 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f54731064e4948fdbca8cf87ae2a9533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:31.240512 1704 raft_consensus.cc:2798] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.240600 1704 raft_consensus.cc:491] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.240746 1704 raft_consensus.cc:3054] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.241690 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f54731064e4948fdbca8cf87ae2a9533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.242321 1427 leader_election.cc:290] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.236862 1178 ts_tablet_manager.cc:1428] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.038s user 0.001s sys 0.002s
I20250112 20:51:31.244120 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.244608 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.245390 1178 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.245036 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:31.246213 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet aa9219fc971c4d15b0d383b20b61e540 because the tablet was not found. No further retry: Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:31.247210 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.248634 1185 tablet_replica.cc:331] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.248958 1185 raft_consensus.cc:2238] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.249167 1185 raft_consensus.cc:2267] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.249713 1185 ts_tablet_manager.cc:1905] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:31.249919 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet aa9219fc971c4d15b0d383b20b61e540 because the tablet was not found. No further retry: Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
W20250112 20:51:31.250267 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c697bba271f4455956b9a2511155b79 consensus state has no leader
I20250112 20:51:31.251365 1178 tablet_bootstrap.cc:654] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.256256 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_44b848d13efa4d09b2337d84f8abde75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:31.257169 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_77909cafdcc948d7804b6f8b05ad48e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:31.257742 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f4cae067df1a40098f81cfbd17224c75
I20250112 20:51:31.259073 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4c3f359655c94f1d905943069548845b
W20250112 20:51:31.261551 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:36661 (request call id 75) took 812 ms (client timeout 1000 ms). Trace:
I20250112 20:51:31.261916 1704 raft_consensus.cc:513] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.263082 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.263319 1045 raft_consensus.cc:3054] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.261806 453 rpcz_store.cc:269] 0112 20:51:30.448955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:30.462504 (+ 13549us) service_pool.cc:225] Handling call
0112 20:51:30.503765 (+ 41261us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14f96b8 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:31.261514 (+757749us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:31.178474 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:31.178617 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:31.178625 (+ 8us) write_op.cc:435] Acquired schema lock
0112 20:51:31.178630 (+ 5us) tablet.cc:659] Decoding operations
0112 20:51:31.178740 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:31.178751 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:31.178759 (+ 8us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:31.178818 (+ 59us) tablet.cc:698] Row locks acquired
0112 20:51:31.178822 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:31.178873 (+ 51us) write_op.cc:270] Start()
0112 20:51:31.178905 (+ 32us) write_op.cc:276] Timestamp: P: 1736715091178815 usec, L: 0
0112 20:51:31.178911 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:31.179086 (+ 175us) log.cc:844] Serialized 527 byte log entry
0112 20:51:31.252715 (+ 73629us) op_driver.cc:464] REPLICATION: finished
0112 20:51:31.253568 (+ 853us) write_op.cc:301] APPLY: starting
0112 20:51:31.253577 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:31.253604 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:31.253647 (+ 43us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:31.253676 (+ 29us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:31.253706 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:31.253713 (+ 7us) write_op.cc:312] APPLY: finished
0112 20:51:31.253755 (+ 42us) log.cc:844] Serialized 23 byte log entry
0112 20:51:31.253786 (+ 31us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:31.253795 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:31.253799 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:31.253851 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:31.253855 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"spinlock_wait_cycles":55907712,"child_traces":[["op",{"apply.queue_time_us":823,"apply.run_cpu_time_us":279,"apply.run_wall_time_us":279,"mutex_wait_us":38110,"num_ops":1,"prepare.queue_time_us":38645,"prepare.run_cpu_time_us":673,"prepare.run_wall_time_us":674,"replication_time_us":73772}]]}
I20250112 20:51:31.265345 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c378f8f7fc9c4eb8bb4b30be4300eeb2" } new_table_name: "default.Table_e754c35aa13145f18490d18de677c33d" modify_external_catalogs: true
I20250112 20:51:31.265985 1045 raft_consensus.cc:2463] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.266562 847 leader_election.cc:304] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.266832 1430 raft_consensus.cc:2798] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.267009 1430 raft_consensus.cc:695] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.267273 1430 consensus_queue.cc:237] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.267377 1700 raft_consensus.cc:513] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.267961 1700 leader_election.cc:290] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.268958 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.269153 1045 raft_consensus.cc:3054] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.270639 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:31.270215 1178 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.271770 1178 ts_tablet_manager.cc:1397] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20250112 20:51:31.272768 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:31.273207 1178 raft_consensus.cc:357] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.274679 1045 raft_consensus.cc:2463] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.276880 1178 raft_consensus.cc:383] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.275113 847 leader_election.cc:304] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.274237 1704 leader_election.cc:290] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.277511 1701 raft_consensus.cc:2798] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.277673 1701 raft_consensus.cc:695] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.277765 1178 raft_consensus.cc:738] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.277936 1701 consensus_queue.cc:237] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.278528 1178 consensus_queue.cc:260] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.280037 1613 raft_consensus.cc:491] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.280083 1178 ts_tablet_manager.cc:1428] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:31.280222 1613 raft_consensus.cc:513] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:31.280387 1178 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.281000 1613 leader_election.cc:290] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.281862 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.282130 1178 tablet_bootstrap.cc:654] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.282218 911 raft_consensus.cc:2388] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:31.285166 1613 raft_consensus.cc:491] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.285333 1613 raft_consensus.cc:513] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.285784 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.285984 1613 leader_election.cc:290] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:31.286790 983 leader_election.cc:343] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9a692cc5fb6f4afc854c4821274edfdb
I20250112 20:51:31.290350 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.290763 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.297446 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.297693 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.297905 1186 tablet_replica.cc:331] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:31.298179 1186 raft_consensus.cc:2238] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:31.298535 1186 pending_rounds.cc:62] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:31.297724 1178 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.298619 1186 pending_rounds.cc:69] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585011209097216 op_type: NO_OP noop_request { }
I20250112 20:51:31.298772 1186 raft_consensus.cc:2883] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:31.298815 1178 ts_tablet_manager.cc:1397] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20250112 20:51:31.298887 1186 raft_consensus.cc:2267] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.300261 1186 ts_tablet_manager.cc:1905] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.307888 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7a54ce78854137ab2d12e9ce055565" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.308733 1186 ts_tablet_manager.cc:1918] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:31.309273 983 leader_election.cc:304] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.310046 1613 raft_consensus.cc:3054] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.311669 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7a54ce78854137ab2d12e9ce055565" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.317127 1354 raft_consensus.cc:491] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.317226 1186 log.cc:1198] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8b414be57990432bb786daf5faf0cc2e
I20250112 20:51:31.317317 1354 raft_consensus.cc:513] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.317795 1186 ts_tablet_manager.cc:1939] T 8b414be57990432bb786daf5faf0cc2e P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:31.318037 1354 leader_election.cc:290] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.316862 1178 raft_consensus.cc:357] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.318445 1178 raft_consensus.cc:383] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.318514 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.318619 1178 raft_consensus.cc:738] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.318796 913 raft_consensus.cc:2388] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:31.318958 1178 consensus_queue.cc:260] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.319478 911 raft_consensus.cc:2463] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.319731 1178 ts_tablet_manager.cc:1428] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250112 20:51:31.320088 1178 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.320870 1613 raft_consensus.cc:2743] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.322352 1178 tablet_bootstrap.cc:654] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.324537 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:31.325582 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44b848d13efa4d09b2337d84f8abde75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.326889 451 catalog_manager.cc:6596] ReplaceTablet: tablet f4cae067df1a40098f81cfbd17224c75 deleted and replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:31.327294 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c0e48e09ccab48a9a2cf6aa4212f1658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.327847 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:31.328898 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77909cafdcc948d7804b6f8b05ad48e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.330140 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3407e578085546e59a593ea9e2016ebf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:31.330657 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3407e578085546e59a593ea9e2016ebf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.331854 448 catalog_manager.cc:6596] ReplaceTablet: tablet d770e0399e80491f8de9b08d1eb2f139 deleted and replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:31.337306 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.337638 1186 tablet_replica.cc:331] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:31.337890 1186 raft_consensus.cc:2238] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.338142 1186 raft_consensus.cc:2267] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.338762 1186 ts_tablet_manager.cc:1905] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.266059 1164 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.340325 1164 ts_tablet_manager.cc:1397] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.337s user 0.027s sys 0.070s
I20250112 20:51:31.341452 1164 raft_consensus.cc:357] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.341714 1164 raft_consensus.cc:383] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.341820 1164 raft_consensus.cc:738] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.342103 1164 consensus_queue.cc:260] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.342782 1164 ts_tablet_manager.cc:1428] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.343120 1164 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.297590 1427 raft_consensus.cc:491] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.344913 1427 raft_consensus.cc:513] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.345651 1427 leader_election.cc:290] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.347286 1164 tablet_bootstrap.cc:654] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.351840 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.353355 882 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.354070 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_39c63dc2cbe1434f83e4872d1fba7757" } new_table_name: "default.Table_30fc56d0c044444a9a07861b6650fac6" modify_external_catalogs: true
I20250112 20:51:31.358883 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.359102 911 raft_consensus.cc:3054] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.362066 911 raft_consensus.cc:2463] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:31.363616 983 leader_election.cc:343] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: cc971026f6b44db6af8215890ba71075
I20250112 20:51:31.363787 983 leader_election.cc:304] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.364059 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.364323 1354 raft_consensus.cc:2743] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.365275 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.365571 775 raft_consensus.cc:2463] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.368453 753 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.369824 1164 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.370052 1164 ts_tablet_manager.cc:1397] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20250112 20:51:31.371085 1164 raft_consensus.cc:357] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.371366 1164 raft_consensus.cc:383] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.371506 1164 raft_consensus.cc:738] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.371852 1164 consensus_queue.cc:260] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.372665 1164 ts_tablet_manager.cc:1428] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:31.373067 1164 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.374291 1186 ts_tablet_manager.cc:1918] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.376379 1164 tablet_bootstrap.cc:654] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.383671 1186 log.cc:1198] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ef89229192934339954566e366b6891c
I20250112 20:51:31.384157 1186 ts_tablet_manager.cc:1939] T ef89229192934339954566e366b6891c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:31.385411 1164 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.386219 1164 ts_tablet_manager.cc:1397] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20250112 20:51:31.387504 1164 raft_consensus.cc:357] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.388038 1184 raft_consensus.cc:2238] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.388445 1184 pending_rounds.cc:62] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:31.388548 1184 pending_rounds.cc:69] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585009583968256 op_type: NO_OP noop_request { }
I20250112 20:51:31.388700 1184 raft_consensus.cc:2883] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:31.388870 1184 raft_consensus.cc:2267] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.389057 1164 raft_consensus.cc:383] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.389394 1164 raft_consensus.cc:738] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.393477 1164 consensus_queue.cc:260] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.394260 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.394524 775 raft_consensus.cc:2463] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.396015 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaa5246382745ecbea4b9bd4825e8f1" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.396240 775 raft_consensus.cc:3054] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.399116 775 raft_consensus.cc:2463] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.400312 1164 ts_tablet_manager.cc:1428] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
W20250112 20:51:31.403649 1696 consensus_queue.cc:1159] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250112 20:51:31.404471 1184 ts_tablet_manager.cc:1905] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.406364 1164 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.408809 1164 tablet_bootstrap.cc:654] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.412130 1184 ts_tablet_manager.cc:1918] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:31.412313 1184 log.cc:1198] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:31.412753 1184 ts_tablet_manager.cc:1939] T 2afa5b0717df4f83a2b44efac12fc06b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:31.414063 1184 tablet_replica.cc:331] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:31.414322 1184 raft_consensus.cc:2238] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.414530 1184 raft_consensus.cc:2267] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.415078 1184 ts_tablet_manager.cc:1905] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.421372 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45484f8e48524b86a6f418b88a71ac18" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.421590 1164 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.421648 773 raft_consensus.cc:3054] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.422350 1164 ts_tablet_manager.cc:1397] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20250112 20:51:31.423723 1164 raft_consensus.cc:357] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.425383 1164 raft_consensus.cc:383] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.425663 1164 raft_consensus.cc:738] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.427054 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.428081 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.430226 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 70653110aec74727acec9d63c2eb0f05 (DEFAULT_TABLE table=default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.430584 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70653110aec74727acec9d63c2eb0f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.425997 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.426698 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.428848 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.429563 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc920e92de545fc961f82e7d0de28cd (DEFAULT_TABLE table=default.Table_211ca26c30e94a03aaf75698f6765585 [id=9dda94626ac842fabfa576cfb417dac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.433717 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc920e92de545fc961f82e7d0de28cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.425108 773 raft_consensus.cc:2463] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.438050 710 leader_election.cc:304] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.438328 1663 raft_consensus.cc:2798] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.438448 1663 raft_consensus.cc:491] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.438534 1663 raft_consensus.cc:3054] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.442420 1663 raft_consensus.cc:513] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.442893 1164 consensus_queue.cc:260] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.443248 1663 leader_election.cc:290] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.443608 710 leader_election.cc:304] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.444157 1427 raft_consensus.cc:2798] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.444273 1427 raft_consensus.cc:695] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.444520 1427 consensus_queue.cc:237] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.446941 1164 ts_tablet_manager.cc:1428] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250112 20:51:31.447913 1164 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.451344 1164 tablet_bootstrap.cc:654] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.454303 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:31.454736 1044 raft_consensus.cc:3054] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.455143 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a9c6d59d5be4e2f9c46487c4f674ad7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.455102 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a90f801d854d4d8f7f09112e2d2929" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.455330 773 raft_consensus.cc:2388] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:31.458652 1430 consensus_queue.cc:1035] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.460971 1663 raft_consensus.cc:491] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.461187 1663 raft_consensus.cc:513] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.462059 1663 leader_election.cc:290] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.466228 1663 raft_consensus.cc:491] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.466415 1663 raft_consensus.cc:513] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.467097 1663 leader_election.cc:290] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.471704 1164 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.472682 1164 ts_tablet_manager.cc:1397] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
I20250112 20:51:31.479449 882 tablet_service.cc:1467] Processing CreateTablet for tablet 70653110aec74727acec9d63c2eb0f05 (DEFAULT_TABLE table=default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.479882 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70653110aec74727acec9d63c2eb0f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.482882 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.484370 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7a54ce78854137ab2d12e9ce055565" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.485596 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7a54ce78854137ab2d12e9ce055565" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.485819 910 raft_consensus.cc:3054] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.487058 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3f359655c94f1d905943069548845b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.487782 450 catalog_manager.cc:5526] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.488250 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:31.488571 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]). Expected version 1 got 0
I20250112 20:51:31.488651 910 raft_consensus.cc:2463] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:31.489005 983 leader_election.cc:304] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.489444 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.489516 983 leader_election.cc:304] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:31.489967 983 leader_election.cc:343] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:31.490103 983 leader_election.cc:304] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:31.490212 710 leader_election.cc:343] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:31.490275 983 raft_consensus.cc:2681] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:31.490525 1354 raft_consensus.cc:3054] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.490523 710 leader_election.cc:343] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.490758 1163 tablet_bootstrap.cc:492] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:31.490815 710 leader_election.cc:343] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.490952 1163 ts_tablet_manager.cc:1397] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.384s user 0.040s sys 0.081s
W20250112 20:51:31.490998 710 leader_election.cc:343] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:31.491627 710 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:31.491803 710 leader_election.cc:343] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.492287 1163 raft_consensus.cc:357] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.492476 1163 raft_consensus.cc:383] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.492558 1163 raft_consensus.cc:738] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.492748 1163 consensus_queue.cc:260] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.493086 1164 raft_consensus.cc:357] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.493338 1164 raft_consensus.cc:383] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.493448 1164 raft_consensus.cc:738] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.493670 1164 consensus_queue.cc:260] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.494226 1164 ts_tablet_manager.cc:1428] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250112 20:51:31.458686 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c5469ad8d235490ebfbbb933c659d6ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:31.494511 1164 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:31.494860 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5469ad8d235490ebfbbb933c659d6ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.495959 1164 tablet_bootstrap.cc:654] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.496847 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.498502 1430 consensus_queue.cc:1035] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.459362 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6c73da0cad5b49beb8a7ec45996a6d6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:31.500380 1185 ts_tablet_manager.cc:1918] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.500522 1185 log.cc:1198] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7f39195945024f4fba9942faec70bc31
W20250112 20:51:31.500576 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c73da0cad5b49beb8a7ec45996a6d6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.501113 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:31.498937 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3f359655c94f1d905943069548845b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.501538 1701 consensus_queue.cc:1035] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 20:51:31.501449 450 catalog_manager.cc:5526] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.502183 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:31.502446 1178 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.502509 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 1 got 0
W20250112 20:51:31.502501 983 leader_election.cc:343] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 9a692cc5fb6f4afc854c4821274edfdb
I20250112 20:51:31.502604 1178 ts_tablet_manager.cc:1397] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.183s user 0.021s sys 0.021s
I20250112 20:51:31.502635 983 leader_election.cc:304] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.502672 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:31.503573 1178 raft_consensus.cc:357] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.503856 1178 raft_consensus.cc:383] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.503966 1178 raft_consensus.cc:738] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.504199 1178 consensus_queue.cc:260] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.504994 1178 ts_tablet_manager.cc:1428] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.505386 1178 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.507997 1178 tablet_bootstrap.cc:654] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:31.509776 710 consensus_peers.cc:487] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef89229192934339954566e366b6891c. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:31.509982 710 leader_election.cc:343] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.510131 710 leader_election.cc:304] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.510918 1185 ts_tablet_manager.cc:1939] T 7f39195945024f4fba9942faec70bc31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:31.511613 1354 raft_consensus.cc:2743] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.511852 1354 raft_consensus.cc:2743] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.512259 1427 raft_consensus.cc:2743] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.512279 1354 raft_consensus.cc:2743] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.512693 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.513092 1185 tablet_replica.cc:331] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.513284 1185 raft_consensus.cc:2238] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:31.513538 1185 pending_rounds.cc:62] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:31.513590 1185 pending_rounds.cc:69] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585010460082176 op_type: NO_OP noop_request { }
I20250112 20:51:31.513700 1185 raft_consensus.cc:2883] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:31.513803 1185 raft_consensus.cc:2267] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.514364 1185 ts_tablet_manager.cc:1905] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:31.515810 710 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.515940 710 leader_election.cc:304] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.516726 1427 raft_consensus.cc:2743] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:31.460851 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4303e4aec9694c49aca7c1532eaab3ed consensus state has no leader
W20250112 20:51:31.517241 710 leader_election.cc:343] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.467594 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ef68d20727c04b09980aac02d08a81bb" } new_table_name: "default.Table_8eeab516a6794571a8336068218e7a3a" modify_external_catalogs: true
I20250112 20:51:31.517380 710 leader_election.cc:304] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.518141 1427 raft_consensus.cc:2743] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.468012 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a374fefd5800476d8e57e23cf0577a2a" } modify_external_catalogs: true
I20250112 20:51:31.518524 710 leader_election.cc:304] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.518656 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a374fefd5800476d8e57e23cf0577a2a" } modify_external_catalogs: true
I20250112 20:51:31.519131 1427 raft_consensus.cc:2798] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.519264 1427 raft_consensus.cc:695] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.519506 1427 consensus_queue.cc:237] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.519762 1178 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.519940 1178 ts_tablet_manager.cc:1397] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20250112 20:51:31.517297 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4fb0c297ce434ae4be2c6eba23a061a6" } modify_external_catalogs: true
I20250112 20:51:31.520143 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4fb0c297ce434ae4be2c6eba23a061a6" } modify_external_catalogs: true
I20250112 20:51:31.520889 1178 raft_consensus.cc:357] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.521201 1178 raft_consensus.cc:383] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.468448 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_98c777cc5e4149168e22f0635ec52fbd" } new_table_name: "default.Table_986d0d39b1fa41abb2166e064df20564" modify_external_catalogs: true
I20250112 20:51:31.521322 1178 raft_consensus.cc:738] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.521590 1178 consensus_queue.cc:260] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.522200 1178 ts_tablet_manager.cc:1428] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:31.468840 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 48e390eb2e404250a8b6edba2ee4f6d6 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:31.523038 1178 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:31.469743 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4561956c97824824a55283a8da21929e has no consensus state
I20250112 20:51:31.525141 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 7bd84ab3a8fc4580b971d6b9adb18d6f from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:31.528081 1354 raft_consensus.cc:491] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.528797 1184 ts_tablet_manager.cc:1918] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.534912 1178 tablet_bootstrap.cc:654] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.535480 1185 ts_tablet_manager.cc:1918] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:31.535630 1185 log.cc:1198] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ef89229192934339954566e366b6891c
I20250112 20:51:31.536033 1185 ts_tablet_manager.cc:1939] T ef89229192934339954566e366b6891c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:31.528412 1354 raft_consensus.cc:513] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:31.541383 1185 tablet_replica.cc:331] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.541877 1185 raft_consensus.cc:2238] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.542376 1185 raft_consensus.cc:2267] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.543154 1185 ts_tablet_manager.cc:1905] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.545214 1163 ts_tablet_manager.cc:1428] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.037s user 0.003s sys 0.011s
I20250112 20:51:31.545759 1163 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.547405 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c55b922c39545ac9c89115cd587ea8c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.551328 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu with pid 563
I20250112 20:51:31.552724 1163 tablet_bootstrap.cc:654] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:31.554672 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 51376.7us
I20250112 20:51:31.555953 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.556555 1354 leader_election.cc:290] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.557029 1184 log.cc:1198] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e0f1c76db0d64624bd43f0e538f0649d
I20250112 20:51:31.557901 1184 ts_tablet_manager.cc:1939] T e0f1c76db0d64624bd43f0e538f0649d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:31.559041 1184 tablet_replica.cc:331] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:31.559180 751 tablet_service.cc:1467] Processing CreateTablet for tablet 70653110aec74727acec9d63c2eb0f05 (DEFAULT_TABLE table=default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.559648 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70653110aec74727acec9d63c2eb0f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.569845 1178 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.570113 1178 ts_tablet_manager.cc:1397] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20250112 20:51:31.571185 1178 raft_consensus.cc:357] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.571537 1178 raft_consensus.cc:383] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.571708 1178 raft_consensus.cc:738] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.572060 1178 consensus_queue.cc:260] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.572969 1178 ts_tablet_manager.cc:1428] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:31.573616 1178 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.574081 882 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b788acf8047acb34ca2596bbf141b (DEFAULT_TABLE table=default.Table_31a6df3a24c042a38ee31d789e5f46eb [id=22e212eacf0449b48069d0c9b7cda461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.574177 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b788acf8047acb34ca2596bbf141b (DEFAULT_TABLE table=default.Table_31a6df3a24c042a38ee31d789e5f46eb [id=22e212eacf0449b48069d0c9b7cda461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.574687 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7b788acf8047acb34ca2596bbf141b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:31.580471 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bbb679585fc44699553fe85ea044566 consensus state has no leader
I20250112 20:51:31.562170 1044 raft_consensus.cc:3054] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.581053 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7b788acf8047acb34ca2596bbf141b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.583842 1427 raft_consensus.cc:491] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.584038 1427 raft_consensus.cc:513] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.584179 1587 consensus_queue.cc:1035] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.584650 1178 tablet_bootstrap.cc:654] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.584682 1427 leader_election.cc:290] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.568305 1163 tablet_bootstrap.cc:492] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.585206 1163 ts_tablet_manager.cc:1397] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.001s
I20250112 20:51:31.585278 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.585570 910 raft_consensus.cc:2463] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.586030 710 leader_election.cc:304] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.586050 1163 raft_consensus.cc:357] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.586261 1163 raft_consensus.cc:383] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.586336 1163 raft_consensus.cc:738] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.586424 1427 raft_consensus.cc:2798] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.586503 1427 raft_consensus.cc:491] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.586591 1427 raft_consensus.cc:3054] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.586617 1163 consensus_queue.cc:260] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.587157 1163 ts_tablet_manager.cc:1428] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:31.587437 1163 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.589383 1427 raft_consensus.cc:513] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.589584 1163 tablet_bootstrap.cc:654] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.590440 1427 leader_election.cc:290] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.590507 1184 raft_consensus.cc:2238] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.590709 1184 raft_consensus.cc:2267] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.591231 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.591369 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.572846 983 leader_election.cc:304] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.573084 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b788acf8047acb34ca2596bbf141b (DEFAULT_TABLE table=default.Table_31a6df3a24c042a38ee31d789e5f46eb [id=22e212eacf0449b48069d0c9b7cda461]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:31.591812 710 leader_election.cc:343] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.592142 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7b788acf8047acb34ca2596bbf141b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:31.592315 710 leader_election.cc:343] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.592675 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.592954 910 raft_consensus.cc:3054] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.596275 910 raft_consensus.cc:2463] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:31.596516 1587 raft_consensus.cc:491] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.596752 1587 raft_consensus.cc:513] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.596935 1720 raft_consensus.cc:3054] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.597616 1587 leader_election.cc:290] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.572249 1354 raft_consensus.cc:491] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.598416 1354 raft_consensus.cc:513] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.599241 1354 leader_election.cc:290] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.599467 1178 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.599613 1178 ts_tablet_manager.cc:1397] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.004s
I20250112 20:51:31.599769 710 leader_election.cc:304] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.600230 1427 raft_consensus.cc:2798] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.600391 1178 raft_consensus.cc:357] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.600464 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.600518 1427 raft_consensus.cc:695] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.600589 1178 raft_consensus.cc:383] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.600627 1045 raft_consensus.cc:2463] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.600659 1178 raft_consensus.cc:738] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.600000 1720 raft_consensus.cc:2743] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.601121 847 leader_election.cc:304] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.600845 1427 consensus_queue.cc:237] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.601411 1587 raft_consensus.cc:2798] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.601495 1587 raft_consensus.cc:491] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 20:51:31.601469 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 44063us
I20250112 20:51:31.601655 1587 raft_consensus.cc:3054] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.602047 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.591315 1184 ts_tablet_manager.cc:1905] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.602782 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.602989 774 raft_consensus.cc:2463] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:31.604456 983 leader_election.cc:304] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.604847 1720 raft_consensus.cc:2798] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.605010 1720 raft_consensus.cc:491] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.600953 1178 consensus_queue.cc:260] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.605180 1720 raft_consensus.cc:3054] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.605528 1178 ts_tablet_manager.cc:1428] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:31.606065 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.606321 775 raft_consensus.cc:2463] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.606926 1178 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.607663 1720 raft_consensus.cc:513] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.608390 1720 leader_election.cc:290] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.608629 1178 tablet_bootstrap.cc:654] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.606539 1587 raft_consensus.cc:513] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.609764 881 tablet_service.cc:1467] Processing CreateTablet for tablet 2111e988ea634e999a1bbbf7b270153b (DEFAULT_TABLE table=default.Table_14bd18a76fc94d41b11ad4252e4c491f [id=465efda4e0904046a3a7ee00a4e373e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.609987 911 raft_consensus.cc:2388] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:31.610211 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2111e988ea634e999a1bbbf7b270153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.610607 887 tablet_service.cc:1467] Processing CreateTablet for tablet bc05d287d81c46558acdd9a42edda6f8 (DEFAULT_TABLE table=default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.610680 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2111e988ea634e999a1bbbf7b270153b (DEFAULT_TABLE table=default.Table_14bd18a76fc94d41b11ad4252e4c491f [id=465efda4e0904046a3a7ee00a4e373e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.610975 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc05d287d81c46558acdd9a42edda6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.611140 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2111e988ea634e999a1bbbf7b270153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.617018 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.617331 910 raft_consensus.cc:2388] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:31.617703 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.618003 775 raft_consensus.cc:3054] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.618127 1587 leader_election.cc:290] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.618816 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.554468 1185 ts_tablet_manager.cc:1918] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.555590 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a8f71233d04ca39e49932308e56492" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.619580 1185 log.cc:1198] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b32dc02497f64a2e9ad99da6c9115d5d
I20250112 20:51:31.619714 772 raft_consensus.cc:2388] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:31.620046 1185 ts_tablet_manager.cc:1939] T b32dc02497f64a2e9ad99da6c9115d5d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:31.626647 1178 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.627867 1185 tablet_replica.cc:331] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.629472 881 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd (DEFAULT_TABLE table=default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.630169 751 tablet_service.cc:1467] Processing CreateTablet for tablet bc05d287d81c46558acdd9a42edda6f8 (DEFAULT_TABLE table=default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.630637 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc05d287d81c46558acdd9a42edda6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.631006 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1a4b5ac2f94874b99b3ef790a034fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.632207 753 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd (DEFAULT_TABLE table=default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.632615 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1a4b5ac2f94874b99b3ef790a034fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.630673 1185 raft_consensus.cc:2238] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.637076 1185 raft_consensus.cc:2267] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.638159 1185 ts_tablet_manager.cc:1905] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.638252 1178 ts_tablet_manager.cc:1397] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250112 20:51:31.642882 753 tablet_service.cc:1467] Processing CreateTablet for tablet aa9219fc971c4d15b0d383b20b61e540 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.643429 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9219fc971c4d15b0d383b20b61e540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.647308 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2111e988ea634e999a1bbbf7b270153b (DEFAULT_TABLE table=default.Table_14bd18a76fc94d41b11ad4252e4c491f [id=465efda4e0904046a3a7ee00a4e373e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.647744 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2111e988ea634e999a1bbbf7b270153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.647758 1018 tablet_service.cc:1467] Processing CreateTablet for tablet bc05d287d81c46558acdd9a42edda6f8 (DEFAULT_TABLE table=default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.648947 887 tablet_service.cc:1467] Processing CreateTablet for tablet aa9219fc971c4d15b0d383b20b61e540 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.649403 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9219fc971c4d15b0d383b20b61e540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.649788 1178 raft_consensus.cc:357] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.649999 1178 raft_consensus.cc:383] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.650238 1178 raft_consensus.cc:738] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.650485 1178 consensus_queue.cc:260] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.655733 1178 ts_tablet_manager.cc:1428] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250112 20:51:31.655998 1178 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.656589 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 54ab0ae10f09469fa4f913e049002481
I20250112 20:51:31.657158 1024 tablet_service.cc:1467] Processing CreateTablet for tablet aa9219fc971c4d15b0d383b20b61e540 (DEFAULT_TABLE table=default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.657680 1178 tablet_bootstrap.cc:654] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.657761 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc05d287d81c46558acdd9a42edda6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.658070 983 leader_election.cc:304] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.658838 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbbefa7b55041a6a6b50b3c96c45833" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.665612 1663 raft_consensus.cc:491] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.665757 1663 raft_consensus.cc:513] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.666349 1663 leader_election.cc:290] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.666503 1663 consensus_queue.cc:1035] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.667603 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.667860 911 raft_consensus.cc:2463] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.668437 710 leader_election.cc:304] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.668797 1663 raft_consensus.cc:2798] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.668918 1663 raft_consensus.cc:491] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.669054 1663 raft_consensus.cc:3054] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.671727 1663 raft_consensus.cc:513] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.672398 1663 leader_election.cc:290] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.672842 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.673063 911 raft_consensus.cc:3054] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.673871 1663 raft_consensus.cc:491] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.674289 1663 raft_consensus.cc:513] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.674818 1663 leader_election.cc:290] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.676169 911 raft_consensus.cc:2463] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:31.676892 710 leader_election.cc:304] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:31.677011 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd (DEFAULT_TABLE table=default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.677187 1427 raft_consensus.cc:2798] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.677403 1427 raft_consensus.cc:695] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.677778 1427 consensus_queue.cc:237] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.678284 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:31.678768 710 leader_election.cc:343] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.679176 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:31.679694 710 leader_election.cc:343] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.679824 710 leader_election.cc:304] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.680090 1663 raft_consensus.cc:2743] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.684986 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.685371 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:31.685921 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f4cae067df1a40098f81cfbd17224c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cae067df1a40098f81cfbd17224c75 already in progress: creating tablet
I20250112 20:51:31.686694 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9219fc971c4d15b0d383b20b61e540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.687016 1045 raft_consensus.cc:2388] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
W20250112 20:51:31.691318 710 leader_election.cc:343] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:31.691998 710 leader_election.cc:343] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.692760 847 leader_election.cc:304] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.693193 1720 raft_consensus.cc:2743] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.693244 1587 raft_consensus.cc:2743] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.693560 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1a4b5ac2f94874b99b3ef790a034fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.690218 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:31.694461 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8b414be57990432bb786daf5faf0cc2e (table default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]) successfully deleted
I20250112 20:51:31.697204 1430 consensus_queue.cc:1035] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.699258 1163 tablet_bootstrap.cc:492] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.699483 1163 ts_tablet_manager.cc:1397] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.007s
I20250112 20:51:31.701087 1163 raft_consensus.cc:357] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:31.702561 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4c3f359655c94f1d905943069548845b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c3f359655c94f1d905943069548845b already in progress: creating tablet
W20250112 20:51:31.702831 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2afa5b0717df4f83a2b44efac12fc06b because the tablet was not found. No further retry: Not found: Tablet not found: 2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:31.703027 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ef89229192934339954566e366b6891c (table default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]) successfully deleted
W20250112 20:51:31.703264 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet bc05d287d81c46558acdd9a42edda6f8 because the tablet was not found. No further retry: Not found: Tablet not found: bc05d287d81c46558acdd9a42edda6f8
W20250112 20:51:31.703481 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d770e0399e80491f8de9b08d1eb2f139 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d770e0399e80491f8de9b08d1eb2f139 already in progress: creating tablet
I20250112 20:51:31.708868 1163 raft_consensus.cc:383] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.709025 775 raft_consensus.cc:2463] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:31.709089 1184 ts_tablet_manager.cc:1918] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.709240 1184 log.cc:1198] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9a692cc5fb6f4afc854c4821274edfdb
I20250112 20:51:31.710407 1184 ts_tablet_manager.cc:1939] T 9a692cc5fb6f4afc854c4821274edfdb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:31.715889 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db9178cf782d4bd28eba90cc03796885 has no consensus state
I20250112 20:51:31.716286 1720 raft_consensus.cc:491] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.716436 1720 raft_consensus.cc:513] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:31.719460 983 leader_election.cc:343] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: b32dc02497f64a2e9ad99da6c9115d5d
I20250112 20:51:31.719720 1720 leader_election.cc:290] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.721098 1184 tablet_replica.cc:331] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:31.722100 1163 raft_consensus.cc:738] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.722482 1163 consensus_queue.cc:260] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.724092 1163 ts_tablet_manager.cc:1428] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250112 20:51:31.718705 1164 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.729665 1163 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.729714 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 64f1119a719b4c0fa47650dc1fb1c27d (DEFAULT_TABLE table=default.Table_28f7cb2ca8314f618581e50f1e1bc16a [id=d4d0c3a7693540d98823bb8a175a404e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.730024 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 48e390eb2e404250a8b6edba2ee4f6d6
I20250112 20:51:31.730026 1164 ts_tablet_manager.cc:1397] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.235s user 0.028s sys 0.037s
I20250112 20:51:31.729936 983 leader_election.cc:304] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.730283 1720 raft_consensus.cc:2798] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.730450 455 catalog_manager.cc:6596] ReplaceTablet: tablet 48e390eb2e404250a8b6edba2ee4f6d6 deleted and replaced by tablet 8229e4ef771b4032a4bfae3190a851e3
I20250112 20:51:31.730489 1720 raft_consensus.cc:695] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:31.730655 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f1119a719b4c0fa47650dc1fb1c27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.730754 1720 consensus_queue.cc:237] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.731504 1163 tablet_bootstrap.cc:654] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.732797 1164 raft_consensus.cc:357] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.734532 1164 raft_consensus.cc:383] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.734768 1164 raft_consensus.cc:738] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.735517 1164 consensus_queue.cc:260] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.737567 1178 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.737579 1184 raft_consensus.cc:2238] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.737809 1178 ts_tablet_manager.cc:1397] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.013s
I20250112 20:51:31.738724 1178 raft_consensus.cc:357] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.739058 1178 raft_consensus.cc:383] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.739168 1178 raft_consensus.cc:738] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.739444 1178 consensus_queue.cc:260] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.741900 1178 ts_tablet_manager.cc:1428] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:31.743237 1184 raft_consensus.cc:2267] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.743855 1164 ts_tablet_manager.cc:1428] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 20:51:31.743853 1184 ts_tablet_manager.cc:1905] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.744342 1178 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:31.744946 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:45153 (error 108) [suppressed 1 similar messages]
I20250112 20:51:31.745425 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7bd84ab3a8fc4580b971d6b9adb18d6f
I20250112 20:51:31.745862 446 catalog_manager.cc:6596] ReplaceTablet: tablet 7bd84ab3a8fc4580b971d6b9adb18d6f deleted and replaced by tablet fa2f3389061143509354f37ae93ef0c7
I20250112 20:51:31.759951 1184 ts_tablet_manager.cc:1918] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.761406 1163 tablet_bootstrap.cc:492] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.756042 1638 raft_consensus.cc:491] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.764997 1638 raft_consensus.cc:513] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.765231 1164 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.765601 1638 leader_election.cc:290] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.765777 1638 consensus_queue.cc:1035] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.767469 1164 tablet_bootstrap.cc:654] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.771904 882 tablet_service.cc:1467] Processing CreateTablet for tablet 64f1119a719b4c0fa47650dc1fb1c27d (DEFAULT_TABLE table=default.Table_28f7cb2ca8314f618581e50f1e1bc16a [id=d4d0c3a7693540d98823bb8a175a404e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.773211 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f1119a719b4c0fa47650dc1fb1c27d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:31.773880 1739 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 20:51:31.715794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:45153 (local address 127.0.104.2:60369)
0112 20:51:31.773411 (+ 57617us) negotiation.cc:107] Waiting for socket to connect
0112 20:51:31.773531 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57458,"threads_started":1}
I20250112 20:51:31.774586 1178 tablet_bootstrap.cc:654] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.776858 1184 log.cc:1198] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/616b8aa8fd62486c978c275712064f77
I20250112 20:51:31.776918 1163 ts_tablet_manager.cc:1397] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.014s
I20250112 20:51:31.777055 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 61fac2cf43c24c158e340cbc8dd176c3 (DEFAULT_TABLE table=default.Table_eece5bebd4c54ee784339d7cb43daf6e [id=4d672d282c90487b80b232150877549a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.777432 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fac2cf43c24c158e340cbc8dd176c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.778084 1163 raft_consensus.cc:357] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.778365 1163 raft_consensus.cc:383] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.778463 1163 raft_consensus.cc:738] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.778714 1163 consensus_queue.cc:260] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.779301 1163 ts_tablet_manager.cc:1428] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.779623 1163 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:31.780829 1740 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 20:51:31.718441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:45153 (local address 127.0.104.3:34149)
0112 20:51:31.757512 (+ 39071us) negotiation.cc:107] Waiting for socket to connect
0112 20:51:31.757556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0112 20:51:31.760274 (+ 2718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 20:51:31.763823 (+ 3549us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":23438,"thread_start_us":21303,"threads_started":1}
I20250112 20:51:31.780930 1720 raft_consensus.cc:491] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.781066 1720 raft_consensus.cc:513] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.781769 1720 leader_election.cc:290] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.782553 1163 tablet_bootstrap.cc:654] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.784646 1184 ts_tablet_manager.cc:1939] T 616b8aa8fd62486c978c275712064f77 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:31.787631 1744 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 20:51:31.745603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:45153 (local address 127.0.104.1:53227)
0112 20:51:31.787070 (+ 41467us) negotiation.cc:107] Waiting for socket to connect
0112 20:51:31.787228 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41334,"threads_started":1}
I20250112 20:51:31.787799 1185 ts_tablet_manager.cc:1918] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:31.787983 1185 log.cc:1198] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8b414be57990432bb786daf5faf0cc2e
I20250112 20:51:31.788396 1185 ts_tablet_manager.cc:1939] T 8b414be57990432bb786daf5faf0cc2e P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:31.789938 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2235e2a63c4e1192b3260085908445" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.790210 913 raft_consensus.cc:2463] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:31.791133 753 tablet_service.cc:1467] Processing CreateTablet for tablet 64f1119a719b4c0fa47650dc1fb1c27d (DEFAULT_TABLE table=default.Table_28f7cb2ca8314f618581e50f1e1bc16a [id=d4d0c3a7693540d98823bb8a175a404e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:31.791424 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:45153, user_credentials={real_user=slave}} blocked reactor thread for 46514.8us
I20250112 20:51:31.792240 1430 raft_consensus.cc:491] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.792416 1430 raft_consensus.cc:513] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.792598 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:31.793017 1430 leader_election.cc:290] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.793797 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.794282 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32dc02497f64a2e9ad99da6c9115d5d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:31.794375 847 leader_election.cc:343] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.794955 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2235e2a63c4e1192b3260085908445" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.795590 983 leader_election.cc:304] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.795981 1720 raft_consensus.cc:2743] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.799350 887 tablet_service.cc:1467] Processing CreateTablet for tablet 61fac2cf43c24c158e340cbc8dd176c3 (DEFAULT_TABLE table=default.Table_eece5bebd4c54ee784339d7cb43daf6e [id=4d672d282c90487b80b232150877549a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.799789 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fac2cf43c24c158e340cbc8dd176c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.800784 1163 tablet_bootstrap.cc:492] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.800940 1163 ts_tablet_manager.cc:1397] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20250112 20:51:31.801653 1163 raft_consensus.cc:357] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.801918 1163 raft_consensus.cc:383] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.802008 1163 raft_consensus.cc:738] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.802238 1163 consensus_queue.cc:260] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.804198 1178 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.804387 1163 ts_tablet_manager.cc:1428] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:31.804833 1163 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.806813 887 tablet_service.cc:1467] Processing CreateTablet for tablet 54ab0ae10f09469fa4f913e049002481 (DEFAULT_TABLE table=default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.807209 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab0ae10f09469fa4f913e049002481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.808046 1163 tablet_bootstrap.cc:654] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.809793 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 54ab0ae10f09469fa4f913e049002481 (DEFAULT_TABLE table=default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.810120 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab0ae10f09469fa4f913e049002481. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:31.813302 710 leader_election.cc:343] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.814234 710 leader_election.cc:304] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.814482 1638 consensus_queue.cc:1035] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.814870 1638 raft_consensus.cc:2798] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.814942 1638 raft_consensus.cc:491] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.815022 1638 raft_consensus.cc:3054] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.816557 1178 ts_tablet_manager.cc:1397] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.002s
I20250112 20:51:31.817289 1178 raft_consensus.cc:357] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.817485 1178 raft_consensus.cc:383] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.817572 1178 raft_consensus.cc:738] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.817757 1178 consensus_queue.cc:260] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.817786 1638 raft_consensus.cc:513] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.818459 1638 leader_election.cc:290] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:31.818418 751 tablet_service.cc:1467] Processing CreateTablet for tablet 61fac2cf43c24c158e340cbc8dd176c3 (DEFAULT_TABLE table=default.Table_eece5bebd4c54ee784339d7cb43daf6e [id=4d672d282c90487b80b232150877549a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.819200 1178 ts_tablet_manager.cc:1428] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:31.819434 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.819762 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092a31de9bc47e4bae5f5aebb180c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.819958 752 tablet_service.cc:1467] Processing CreateTablet for tablet 54ab0ae10f09469fa4f913e049002481 (DEFAULT_TABLE table=default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.820528 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:31.820835 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153): Couldn't send request to peer 0f5c09e899184de6b770dca873501062. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:31.821074 1178 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.822692 1178 tablet_bootstrap.cc:654] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.824229 1164 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:31.824344 1164 ts_tablet_manager.cc:1397] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.002s
I20250112 20:51:31.825091 1164 raft_consensus.cc:357] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:31.825348 1164 raft_consensus.cc:383] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.825456 1164 raft_consensus.cc:738] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.825726 1164 consensus_queue.cc:260] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:31.826329 1164 ts_tablet_manager.cc:1428] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.826537 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d (DEFAULT_TABLE table=default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:31.826949 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092a31de9bc47e4bae5f5aebb180c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.828675 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.831947 1163 tablet_bootstrap.cc:492] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.832144 1163 ts_tablet_manager.cc:1397] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250112 20:51:31.832947 1163 raft_consensus.cc:357] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.833145 1163 raft_consensus.cc:383] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.833205 1163 raft_consensus.cc:738] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.833408 1163 consensus_queue.cc:260] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.833915 1163 ts_tablet_manager.cc:1428] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:31.834230 1163 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.834292 1178 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.834441 1178 ts_tablet_manager.cc:1397] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250112 20:51:31.835063 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.835160 1178 raft_consensus.cc:357] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.835384 1178 raft_consensus.cc:383] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.835462 1178 raft_consensus.cc:738] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.835675 1178 consensus_queue.cc:260] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.836218 1178 ts_tablet_manager.cc:1428] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:31.836494 1163 tablet_bootstrap.cc:654] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.837142 1178 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.837452 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2235e2a63c4e1192b3260085908445" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:31.837670 910 raft_consensus.cc:3054] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.838881 1178 tablet_bootstrap.cc:654] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.841171 910 raft_consensus.cc:2463] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:31.843237 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2235e2a63c4e1192b3260085908445" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:31.844205 710 leader_election.cc:343] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.844578 710 leader_election.cc:304] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.845264 1745 raft_consensus.cc:2798] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.847417 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.848281 1430 raft_consensus.cc:491] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.848449 1430 raft_consensus.cc:513] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.849129 1430 leader_election.cc:290] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.849699 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.849907 1045 raft_consensus.cc:2463] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.850253 847 leader_election.cc:304] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.850487 1430 raft_consensus.cc:2798] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.850576 1430 raft_consensus.cc:491] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.850667 1430 raft_consensus.cc:3054] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.852474 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.853289 750 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.855583 1430 raft_consensus.cc:513] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.856083 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.856827 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.857074 1045 raft_consensus.cc:3054] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.857093 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.857653 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.858392 750 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.859114 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.859599 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:31.860069 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d770e0399e80491f8de9b08d1eb2f139 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d770e0399e80491f8de9b08d1eb2f139 consensus state has no leader
I20250112 20:51:31.860095 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.860627 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.861202 1045 raft_consensus.cc:2463] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.861693 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:31.861739 847 leader_election.cc:304] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.862033 1587 raft_consensus.cc:2798] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.862145 1587 raft_consensus.cc:695] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.862398 1587 consensus_queue.cc:237] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.865182 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:31.866114 1430 leader_election.cc:290] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.868487 1762 tablet.cc:1721] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:31.872197 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7f39195945024f4fba9942faec70bc31 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:31.872433 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ef89229192934339954566e366b6891c (table default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]) successfully deleted
I20250112 20:51:31.873760 1663 consensus_queue.cc:1035] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.875108 448 catalog_manager.cc:5526] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.875869 448 catalog_manager.cc:5526] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.876348 448 catalog_manager.cc:5526] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.876729 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:31.876988 448 catalog_manager.cc:5526] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.877380 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:31.877737 448 catalog_manager.cc:5526] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.878093 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:31.878393 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:31.878597 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
I20250112 20:51:31.878868 1745 raft_consensus.cc:695] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:31.879201 1745 consensus_queue.cc:237] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.892077 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.892318 776 raft_consensus.cc:2463] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.892691 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.892930 773 raft_consensus.cc:2463] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.893255 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158cfe5ba52c4771af97841885b29f9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:31.893499 772 raft_consensus.cc:2388] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:31.896164 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f1119a719b4c0fa47650dc1fb1c27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.903884 1164 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:31.904350 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fac2cf43c24c158e340cbc8dd176c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.908958 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ab0ae10f09469fa4f913e049002481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.910065 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092a31de9bc47e4bae5f5aebb180c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.914059 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:31.920949 1761 tablet.cc:1721] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:31.922974 1163 tablet_bootstrap.cc:492] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:31.923162 1163 ts_tablet_manager.cc:1397] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.089s user 0.015s sys 0.015s
I20250112 20:51:31.923593 1178 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:31.923730 1178 ts_tablet_manager.cc:1397] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.087s user 0.030s sys 0.030s
I20250112 20:51:31.923905 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:31.924197 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:31.924135 1163 raft_consensus.cc:357] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.924351 1163 raft_consensus.cc:383] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.924445 1163 raft_consensus.cc:738] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.924448 1178 raft_consensus.cc:357] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.924674 1178 raft_consensus.cc:383] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.924664 1163 consensus_queue.cc:260] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.924769 1178 raft_consensus.cc:738] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.925287 1163 ts_tablet_manager.cc:1428] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:31.925645 1163 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:31.926187 1178 consensus_queue.cc:260] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.927235 1178 ts_tablet_manager.cc:1428] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:31.927536 1178 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:31.929237 1178 tablet_bootstrap.cc:654] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.930508 1163 tablet_bootstrap.cc:654] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.924536 446 catalog_manager.cc:5526] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.935813 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]). Expected version 1 got 0
I20250112 20:51:31.936245 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:31.936565 446 catalog_manager.cc:5526] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.937194 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:31.939507 1587 consensus_queue.cc:1035] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.947613 1164 tablet_bootstrap.cc:654] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.951853 1430 raft_consensus.cc:491] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.952142 1430 raft_consensus.cc:513] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.954030 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:31.954265 1045 raft_consensus.cc:2463] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:31.954640 847 leader_election.cc:304] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.954977 1587 raft_consensus.cc:2798] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.955082 1587 raft_consensus.cc:491] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.955176 1587 raft_consensus.cc:3054] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.958400 1185 tablet_replica.cc:331] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:31.958485 1587 raft_consensus.cc:513] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.958662 1185 raft_consensus.cc:2238] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:31.958868 1185 raft_consensus.cc:2267] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:31.959364 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.959564 1048 raft_consensus.cc:3054] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.959731 1185 ts_tablet_manager.cc:1905] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:31.962275 1048 raft_consensus.cc:2463] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.962576 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f11304909b416aa1e9db1abf633a89" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.962790 774 raft_consensus.cc:3054] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.965618 847 leader_election.cc:304] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:31.965974 1704 raft_consensus.cc:2798] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.966166 1704 raft_consensus.cc:695] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:31.966475 1704 consensus_queue.cc:237] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.968988 774 raft_consensus.cc:2463] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:31.970151 1046 raft_consensus.cc:3054] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.970264 1745 raft_consensus.cc:491] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.970459 1745 raft_consensus.cc:513] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.971251 1745 leader_election.cc:290] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:31.972267 847 leader_election.cc:304] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:31.972584 1701 raft_consensus.cc:2798] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.972707 1701 raft_consensus.cc:491] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.873030 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b32dc02497f64a2e9ad99da6c9115d5d (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:31.972800 1701 raft_consensus.cc:3054] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.972954 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 99884.8us
I20250112 20:51:31.974048 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8b414be57990432bb786daf5faf0cc2e (table default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]) successfully deleted
W20250112 20:51:31.974350 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f4cae067df1a40098f81cfbd17224c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cae067df1a40098f81cfbd17224c75 already in progress: creating tablet
I20250112 20:51:31.974558 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 2afa5b0717df4f83a2b44efac12fc06b (table default.Table_ca9e0810a7294c4fab5de64b7fde63af [id=61a7256a26de426ca7f8e0763013ac26]) successfully deleted
I20250112 20:51:31.974895 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e0f1c76db0d64624bd43f0e538f0649d (table default.Table_3137d0bc9d0e41d1b5a6f01daa3e6981 [id=5178ded07b844cf4abc1616c387b8585]) successfully deleted
W20250112 20:51:31.975117 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 953d2ec8a7564c0bbcf0166487dd1746 because the tablet was not found. No further retry: Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:31.975519 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9a692cc5fb6f4afc854c4821274edfdb (table default.Table_1b2cd38a56934793bb10f3381448a3ff [id=cc744300f6a34ce5835090563152954d]) successfully deleted
I20250112 20:51:31.975497 1701 raft_consensus.cc:513] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:31.976261 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 616b8aa8fd62486c978c275712064f77 (table default.Table_cdd9ddf0eb094aec9ef461b1eb2468d3 [id=96ccbfa187ea423bb42dbfbd0fc45e43]) successfully deleted
I20250112 20:51:31.976397 1701 leader_election.cc:290] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:31.976471 1048 raft_consensus.cc:3054] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.976512 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet aa9219fc971c4d15b0d383b20b61e540 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa9219fc971c4d15b0d383b20b61e540 already in progress: creating tablet
I20250112 20:51:31.977138 983 leader_election.cc:304] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:31.977732 1720 raft_consensus.cc:2743] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.979245 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:31.980479 710 leader_election.cc:343] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.981204 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:31.982630 710 leader_election.cc:343] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.982810 710 leader_election.cc:304] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:31.983930 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:31.984625 1745 raft_consensus.cc:2743] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:31.985000 1701 consensus_queue.cc:1035] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.985294 1046 raft_consensus.cc:3054] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.989080 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:31.989354 772 raft_consensus.cc:3054] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:31.985023 847 leader_election.cc:343] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:31.995995 1701 consensus_queue.cc:1035] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:31.976744 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f4cae067df1a40098f81cfbd17224c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cae067df1a40098f81cfbd17224c75 already in progress: creating tablet
W20250112 20:51:31.998070 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4c3f359655c94f1d905943069548845b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c3f359655c94f1d905943069548845b already in progress: creating tablet
W20250112 20:51:31.998342 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet bc05d287d81c46558acdd9a42edda6f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc05d287d81c46558acdd9a42edda6f8 already in progress: creating tablet
I20250112 20:51:32.003562 1587 leader_election.cc:290] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:31.998584 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d770e0399e80491f8de9b08d1eb2f139 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d770e0399e80491f8de9b08d1eb2f139 already in progress: creating tablet
W20250112 20:51:32.004988 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 54ab0ae10f09469fa4f913e049002481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54ab0ae10f09469fa4f913e049002481 already in progress: creating tablet
I20250112 20:51:32.005389 1430 leader_election.cc:290] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:32.005429 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6092a31de9bc47e4bae5f5aebb180c6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092a31de9bc47e4bae5f5aebb180c6d already in progress: creating tablet
I20250112 20:51:32.006162 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b310e4135c2b424fb006a38a60e41066 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:32.007028 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533" } modify_external_catalogs: true
I20250112 20:51:32.007269 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533" } modify_external_catalogs: true
I20250112 20:51:32.008116 1163 tablet_bootstrap.cc:492] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.008365 1163 ts_tablet_manager.cc:1397] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.028s
I20250112 20:51:32.009218 1163 raft_consensus.cc:357] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.009516 1163 raft_consensus.cc:383] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.009639 1163 raft_consensus.cc:738] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.010727 1430 consensus_queue.cc:1035] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.011651 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_196bfe6e0c8e4f50a711232f69d23c87 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 1 got 0
I20250112 20:51:32.011863 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:32.012030 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:32.012212 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:32.012336 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:32.012473 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:32.012722 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:32.014611 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_211ca26c30e94a03aaf75698f6765585" } new_table_name: "default.Table_e6cec6d0054141a6a47a840892189740" modify_external_catalogs: true
I20250112 20:51:32.017119 772 raft_consensus.cc:2463] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.018631 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b2428906915d47bd9fb06b88e971194c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.019186 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2428906915d47bd9fb06b88e971194c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.020341 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_58f5707fbcdd4c339243ab9c0645d2aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.020874 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58f5707fbcdd4c339243ab9c0645d2aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.021956 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6d7de66040254347bff24c5702f6241c" } modify_external_catalogs: true
I20250112 20:51:32.022138 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6d7de66040254347bff24c5702f6241c" } modify_external_catalogs: true
I20250112 20:51:32.023303 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c62b92e5784e44af9e27e902aa48a6ca" } modify_external_catalogs: true
I20250112 20:51:32.023465 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c62b92e5784e44af9e27e902aa48a6ca" } modify_external_catalogs: true
I20250112 20:51:32.023741 449 master_service.cc:946] ReplaceTablet: received request to replace tablet ac4e304a1212412f8e99bfd222a110b5 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:32.024027 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 81b932b7e1e34d4ab18aae40047dc038 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:32.024261 1163 consensus_queue.cc:260] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.024816 1583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:32.025048 1583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:32.027271 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1a04c223889f424e831e82a52a84f804
W20250112 20:51:32.028522 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dc920e92de545fc961f82e7d0de28cd consensus state has no leader
I20250112 20:51:32.030556 1163 ts_tablet_manager.cc:1428] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250112 20:51:32.031589 1163 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.032006 1430 consensus_queue.cc:1035] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.034758 1163 tablet_bootstrap.cc:654] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.036156 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:32.043771 1720 raft_consensus.cc:491] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.043967 1720 raft_consensus.cc:513] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.044723 1720 leader_election.cc:290] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.045658 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.046021 910 raft_consensus.cc:2388] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:32.046821 1745 consensus_queue.cc:1035] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.047335 847 leader_election.cc:304] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:32.047926 1587 raft_consensus.cc:2798] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.048069 1587 raft_consensus.cc:695] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:32.048301 1587 consensus_queue.cc:237] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.049490 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ade9b410d8443249730cacdf7266eb2" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.053524 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:32.053745 772 raft_consensus.cc:3054] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.053949 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:32.054500 454 catalog_manager.cc:6596] ReplaceTablet: tablet 81b932b7e1e34d4ab18aae40047dc038 deleted and replaced by tablet 1fd34e829d0645ea80daaff5d27cb266
I20250112 20:51:32.054991 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 83c13126e53b46c0a1e7beafe1b0d295
I20250112 20:51:32.056511 772 raft_consensus.cc:2463] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.056937 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:32.057451 449 catalog_manager.cc:6596] ReplaceTablet: tablet ac4e304a1212412f8e99bfd222a110b5 deleted and replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:32.062180 983 leader_election.cc:304] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.062688 1720 raft_consensus.cc:3054] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.062991 1587 consensus_queue.cc:1035] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.064702 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5b2a82b2202e4bc6b5164fe9c4f07829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.065410 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.065594 1720 raft_consensus.cc:2743] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.066568 1762 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.070281 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5af0bc83bc415b972cbbe0fb39f75f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.070505 772 raft_consensus.cc:2371] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1. Re-sending same reply.
I20250112 20:51:32.070722 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_284c5bfc11f1458dbade76ac78b87f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.071317 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_284c5bfc11f1458dbade76ac78b87f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.077080 1780 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.085716 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_bfe92375d9a04e5aa7fc64840b0bc050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:32.086272 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 24c8dc85ac9240598d6e413deac19abe (table default.Table_b2ca205a6c6f470c896d5194717dbf72 [id=3a4de1859ffd4db3931ce68febdd8836]). Expected version 2 got 0
W20250112 20:51:32.086462 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfe92375d9a04e5aa7fc64840b0bc050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.086562 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:32.089795 1704 raft_consensus.cc:491] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.089959 1704 raft_consensus.cc:513] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.090829 1704 leader_election.cc:290] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.092600 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.093431 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_29c0f892c9954a598f7f27c4be906a91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.094154 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29c0f892c9954a598f7f27c4be906a91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:32.097958 847 leader_election.cc:343] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:32.098495 847 consensus_peers.cc:487] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b32dc02497f64a2e9ad99da6c9115d5d. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:32.103654 1782 tablet.cc:1721] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.104892 1746 raft_consensus.cc:491] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.105046 1746 raft_consensus.cc:513] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.105698 1746 leader_election.cc:290] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:32.106808 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:32.107281 847 leader_election.cc:343] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.107401 847 leader_election.cc:304] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.107967 1430 raft_consensus.cc:2743] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.108180 1430 consensus_queue.cc:1035] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.110921 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.111157 910 raft_consensus.cc:2463] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:32.110922 1587 consensus_queue.cc:1035] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 20:51:32.112912 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461] alter complete (version 1)
I20250112 20:51:32.113739 710 leader_election.cc:304] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:32.113865 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.115728 1746 raft_consensus.cc:2798] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:32.115875 1746 raft_consensus.cc:491] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:32.115963 1746 raft_consensus.cc:3054] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.118193 1745 raft_consensus.cc:491] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.118319 1745 raft_consensus.cc:513] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.118991 1745 leader_election.cc:290] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:32.120143 1746 raft_consensus.cc:513] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.120381 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_07ca89e09c4f4d28b8d2ab35a14a3d22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:32.120821 1746 leader_election.cc:290] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:32.121158 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07ca89e09c4f4d28b8d2ab35a14a3d22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.121341 1163 tablet_bootstrap.cc:492] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.121506 1163 ts_tablet_manager.cc:1397] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.004s
I20250112 20:51:32.122364 1163 raft_consensus.cc:357] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.122646 1163 raft_consensus.cc:383] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.122725 1163 raft_consensus.cc:738] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.122929 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_16d626d49adf42eea0a98976db7a45b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.123551 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16d626d49adf42eea0a98976db7a45b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.125140 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_196bfe6e0c8e4f50a711232f69d23c87" } new_table_name: "default.Table_458c1d4d727849b88b7502de13ea9a40" modify_external_catalogs: true
I20250112 20:51:32.125308 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f89fdf37a8944ee0be9e90eec745a613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.125747 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f89fdf37a8944ee0be9e90eec745a613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.126361 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90937bd663d44575baa256f844890ba9" } new_table_name: "default.Table_a9245c5997d7416fa75ebf6202856645" modify_external_catalogs: true
I20250112 20:51:32.128000 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3c1abf71e9154b45ba412645d3f22386" } modify_external_catalogs: true
I20250112 20:51:32.128137 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_141fc45760534923b67d082dae64762d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:32.128247 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3c1abf71e9154b45ba412645d3f22386" } modify_external_catalogs: true
W20250112 20:51:32.128592 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_141fc45760534923b67d082dae64762d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.129267 1163 consensus_queue.cc:260] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.131631 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_39bacb349abc4e21ab442b1e55db7a0e" } new_table_name: "default.Table_3b87490cb2634435811d01d9ac40c7ea" modify_external_catalogs: true
I20250112 20:51:32.131682 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a64579137f154510b4d9b3432f9b3403" } modify_external_catalogs: true
I20250112 20:51:32.131839 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a64579137f154510b4d9b3432f9b3403" } modify_external_catalogs: true
W20250112 20:51:32.136852 710 leader_election.cc:343] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.138445 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.138651 1163 ts_tablet_manager.cc:1428] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 20:51:32.138962 1163 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.140702 1163 tablet_bootstrap.cc:654] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.140875 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.151440 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:32.151712 1164 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:32.152009 1164 ts_tablet_manager.cc:1397] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.248s user 0.003s sys 0.006s
I20250112 20:51:32.152801 1164 raft_consensus.cc:357] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.153077 1164 raft_consensus.cc:383] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.153188 1164 raft_consensus.cc:738] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.153476 1164 consensus_queue.cc:260] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:32.155337 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6d5d425e6a493daa46900cbd8db166 consensus state has no leader
I20250112 20:51:32.161264 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8d08048c402345a686d1e3fb3039bbc1
I20250112 20:51:32.162611 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:32.162910 910 raft_consensus.cc:3054] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.165181 1163 tablet_bootstrap.cc:492] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.166159 1163 ts_tablet_manager.cc:1397] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
W20250112 20:51:32.167651 847 consensus_peers.cc:487] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f39195945024f4fba9942faec70bc31. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:32.167505 1163 raft_consensus.cc:357] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.169225 1163 raft_consensus.cc:383] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.169510 1163 raft_consensus.cc:738] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.170054 1163 consensus_queue.cc:260] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.170425 1704 consensus_queue.cc:1035] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250112 20:51:32.173575 910 raft_consensus.cc:2463] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:32.175002 1163 ts_tablet_manager.cc:1428] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:32.176589 1163 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.179669 1163 tablet_bootstrap.cc:654] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.182065 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:32.185091 1178 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.185263 1178 ts_tablet_manager.cc:1397] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.258s user 0.051s sys 0.075s
I20250112 20:51:32.186944 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:32.187033 1163 tablet_bootstrap.cc:492] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.187180 1163 ts_tablet_manager.cc:1397] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20250112 20:51:32.187757 1178 raft_consensus.cc:357] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:32.188024 1178 raft_consensus.cc:383] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.187948 1163 raft_consensus.cc:357] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.188107 1178 raft_consensus.cc:738] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.188148 1163 raft_consensus.cc:383] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.188272 1163 raft_consensus.cc:738] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.188364 1178 consensus_queue.cc:260] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:32.188953 1178 ts_tablet_manager.cc:1428] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:32.189247 1178 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.190027 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:32.188500 1163 consensus_queue.cc:260] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.191357 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.191927 1163 ts_tablet_manager.cc:1428] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:32.192621 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.194442 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f with delete_type TABLET_DATA_DELETED (replaced by tablet fa2f3389061143509354f37ae93ef0c7) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.194667 1186 tablet_replica.cc:331] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.194891 1186 raft_consensus.cc:2238] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.195063 1186 raft_consensus.cc:2267] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.196013 1186 ts_tablet_manager.cc:1905] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.198828 1178 tablet_bootstrap.cc:654] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.201809 1178 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.201957 1178 ts_tablet_manager.cc:1397] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
W20250112 20:51:32.156733 710 leader_election.cc:343] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:32.202518 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 45829.2us
W20250112 20:51:32.202772 710 leader_election.cc:343] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.202842 1178 raft_consensus.cc:357] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.203047 1178 raft_consensus.cc:383] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.203161 1178 raft_consensus.cc:738] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:32.203176 710 leader_election.cc:343] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.203214 1186 ts_tablet_manager.cc:1918] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:32.203298 710 leader_election.cc:304] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:32.203505 1186 log.cc:1198] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7bd84ab3a8fc4580b971d6b9adb18d6f
I20250112 20:51:32.203828 710 leader_election.cc:304] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:32.203949 1746 consensus_queue.cc:1035] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.204142 1186 ts_tablet_manager.cc:1939] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:32.203823 1427 raft_consensus.cc:2743] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.204272 1663 raft_consensus.cc:2798] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.204427 1663 raft_consensus.cc:695] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:32.204735 1164 ts_tablet_manager.cc:1428] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.053s user 0.000s sys 0.001s
I20250112 20:51:32.204705 1663 consensus_queue.cc:237] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.205089 1164 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:32.191437 1185 ts_tablet_manager.cc:1918] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.193514 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.205331 1185 log.cc:1198] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:32.205734 1185 ts_tablet_manager.cc:1939] T d770e0399e80491f8de9b08d1eb2f139 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.203444 1178 consensus_queue.cc:260] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.207026 1761 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.207738 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f with delete_type TABLET_DATA_DELETED (replaced by tablet fa2f3389061143509354f37ae93ef0c7) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.207711 1701 consensus_queue.cc:1035] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.208022 1184 tablet_replica.cc:331] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:32.208217 1184 raft_consensus.cc:2238] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.208534 1184 raft_consensus.cc:2267] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.208879 1163 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.209216 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 903027fe49e34a308108356ba89a9c3a (DEFAULT_TABLE table=default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.209712 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903027fe49e34a308108356ba89a9c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.210758 1163 tablet_bootstrap.cc:654] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.210929 1184 ts_tablet_manager.cc:1905] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.212155 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9b7d9218094a63ad5bad03f0d27eb5 (DEFAULT_TABLE table=default.Table_44b848d13efa4d09b2337d84f8abde75 [id=45162e8ecf744eb38342892200fbe634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.212527 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f9b7d9218094a63ad5bad03f0d27eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.213790 1790 consensus_queue.cc:1035] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.215096 887 tablet_service.cc:1467] Processing CreateTablet for tablet 903027fe49e34a308108356ba89a9c3a (DEFAULT_TABLE table=default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.215468 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903027fe49e34a308108356ba89a9c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.216909 881 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d (DEFAULT_TABLE table=default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.217288 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c11a892641554e4db9bdd558068cac40 (DEFAULT_TABLE table=default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1 [id=e4d8998e4e67471d9951cc9dd7ae5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.217383 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab635fec8a64efd8e0c527d913ccd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.217437 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9b7d9218094a63ad5bad03f0d27eb5 (DEFAULT_TABLE table=default.Table_44b848d13efa4d09b2337d84f8abde75 [id=45162e8ecf744eb38342892200fbe634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.217612 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11a892641554e4db9bdd558068cac40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.217816 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f9b7d9218094a63ad5bad03f0d27eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.219326 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d (DEFAULT_TABLE table=default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.219650 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab635fec8a64efd8e0c527d913ccd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.220610 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2513dfbdccf9426ba98711dae084e5ef (DEFAULT_TABLE table=default.Table_aa4af444eb4c4982a8ea14d673f83647 [id=a6c14d6e73584d5b8901730779020bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.221429 890 tablet_service.cc:1467] Processing CreateTablet for tablet c11a892641554e4db9bdd558068cac40 (DEFAULT_TABLE table=default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1 [id=e4d8998e4e67471d9951cc9dd7ae5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.221567 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2513dfbdccf9426ba98711dae084e5ef (DEFAULT_TABLE table=default.Table_aa4af444eb4c4982a8ea14d673f83647 [id=a6c14d6e73584d5b8901730779020bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.221766 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11a892641554e4db9bdd558068cac40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.221951 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2513dfbdccf9426ba98711dae084e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.224870 1178 ts_tablet_manager.cc:1428] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20250112 20:51:32.214578 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84ab3a8fc4580b971d6b9adb18d6f with delete_type TABLET_DATA_DELETED (replaced by tablet fa2f3389061143509354f37ae93ef0c7) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.214852 749 tablet_service.cc:1467] Processing CreateTablet for tablet 903027fe49e34a308108356ba89a9c3a (DEFAULT_TABLE table=default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.225062 889 tablet_service.cc:1467] Processing CreateTablet for tablet d706f88f14ab4fd9a459944863afaacd (DEFAULT_TABLE table=default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.225386 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 903027fe49e34a308108356ba89a9c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.225405 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d706f88f14ab4fd9a459944863afaacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.226199 1185 tablet_replica.cc:331] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.226413 1185 raft_consensus.cc:2238] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.226564 1185 raft_consensus.cc:2267] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.227108 1185 ts_tablet_manager.cc:1905] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.227516 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2513dfbdccf9426ba98711dae084e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.227972 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d706f88f14ab4fd9a459944863afaacd (DEFAULT_TABLE table=default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.215245 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9b7d9218094a63ad5bad03f0d27eb5 (DEFAULT_TABLE table=default.Table_44b848d13efa4d09b2337d84f8abde75 [id=45162e8ecf744eb38342892200fbe634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.229044 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f9b7d9218094a63ad5bad03f0d27eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.229249 1178 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.231096 1178 tablet_bootstrap.cc:654] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.215485 750 tablet_service.cc:1467] Processing CreateTablet for tablet d706f88f14ab4fd9a459944863afaacd (DEFAULT_TABLE table=default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.234828 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d706f88f14ab4fd9a459944863afaacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.235345 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d706f88f14ab4fd9a459944863afaacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.239027 1185 ts_tablet_manager.cc:1918] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.239193 1185 log.cc:1198] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4c3f359655c94f1d905943069548845b
I20250112 20:51:32.239552 1185 ts_tablet_manager.cc:1939] T 4c3f359655c94f1d905943069548845b P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.239907 1720 raft_consensus.cc:491] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.240083 1720 raft_consensus.cc:513] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.215590 753 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d (DEFAULT_TABLE table=default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.240763 1720 leader_election.cc:290] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.215922 755 tablet_service.cc:1467] Processing CreateTablet for tablet 2513dfbdccf9426ba98711dae084e5ef (DEFAULT_TABLE table=default.Table_aa4af444eb4c4982a8ea14d673f83647 [id=a6c14d6e73584d5b8901730779020bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.215967 756 tablet_service.cc:1467] Processing CreateTablet for tablet c11a892641554e4db9bdd558068cac40 (DEFAULT_TABLE table=default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1 [id=e4d8998e4e67471d9951cc9dd7ae5284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.241467 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.241757 913 raft_consensus.cc:2388] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:32.244513 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab635fec8a64efd8e0c527d913ccd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.244422 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4421d16e3f423794c44970a2469b10" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.244983 983 leader_election.cc:304] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.245411 1720 raft_consensus.cc:3054] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.248106 1720 raft_consensus.cc:2743] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.249038 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2513dfbdccf9426ba98711dae084e5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.253021 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11a892641554e4db9bdd558068cac40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.266855 1704 consensus_queue.cc:1035] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.268254 1178 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.268446 1178 ts_tablet_manager.cc:1397] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.004s
I20250112 20:51:32.269912 1178 raft_consensus.cc:357] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.270548 1178 raft_consensus.cc:383] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.270648 1178 raft_consensus.cc:738] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.270974 1178 consensus_queue.cc:260] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.272836 1178 ts_tablet_manager.cc:1428] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250112 20:51:32.274088 1178 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.275782 1185 tablet_replica.cc:331] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.276116 1185 raft_consensus.cc:2238] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.276489 1185 pending_rounds.cc:62] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:32.276623 1185 pending_rounds.cc:69] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585016340193280 op_type: NO_OP noop_request { }
I20250112 20:51:32.276894 1185 raft_consensus.cc:2883] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:32.277081 1185 raft_consensus.cc:2267] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.278162 1178 tablet_bootstrap.cc:654] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.278720 1185 ts_tablet_manager.cc:1905] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.284224 1184 ts_tablet_manager.cc:1918] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:32.284377 1184 log.cc:1198] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7bd84ab3a8fc4580b971d6b9adb18d6f
I20250112 20:51:32.284878 1184 ts_tablet_manager.cc:1939] T 7bd84ab3a8fc4580b971d6b9adb18d6f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:32.288177 1164 tablet_bootstrap.cc:654] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.289083 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (DEFAULT_TABLE table=default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88 [id=2c1e80f117244d4a91c59578fbc1245f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.289623 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235faeb3c35d46d2882ffcd8cb63ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.292025 882 tablet_service.cc:1467] Processing CreateTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (DEFAULT_TABLE table=default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88 [id=2c1e80f117244d4a91c59578fbc1245f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.292845 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235faeb3c35d46d2882ffcd8cb63ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.292722 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4561956c97824824a55283a8da21929e (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.293496 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4561956c97824824a55283a8da21929e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.293579 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9856e6b625d04f17b0869cee08f69142 (DEFAULT_TABLE table=default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.293553 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4561956c97824824a55283a8da21929e (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.293910 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9856e6b625d04f17b0869cee08f69142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.293903 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4561956c97824824a55283a8da21929e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.294760 1178 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.294919 1178 ts_tablet_manager.cc:1397] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20250112 20:51:32.295861 1178 raft_consensus.cc:357] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.297932 879 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.298341 1186 tablet_replica.cc:331] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.298568 1186 raft_consensus.cc:2238] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.298691 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.298933 1186 raft_consensus.cc:2267] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.299079 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.299057 1018 tablet_service.cc:1467] Processing CreateTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (DEFAULT_TABLE table=default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.299289 876 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.299451 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f6a1e9804e6ab822f9d4c004b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.299464 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.299664 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.299861 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.300165 877 tablet_service.cc:1467] Processing CreateTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (DEFAULT_TABLE table=default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.300351 1186 ts_tablet_manager.cc:1905] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.300472 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f6a1e9804e6ab822f9d4c004b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.300563 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.299690 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.301026 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 9856e6b625d04f17b0869cee08f69142 (DEFAULT_TABLE table=default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.301350 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9856e6b625d04f17b0869cee08f69142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.311770 1186 ts_tablet_manager.cc:1918] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.315933 1186 log.cc:1198] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b32dc02497f64a2e9ad99da6c9115d5d
I20250112 20:51:32.296093 1178 raft_consensus.cc:383] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.318516 1178 raft_consensus.cc:738] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.318835 1178 consensus_queue.cc:260] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.292243 753 tablet_service.cc:1467] Processing CreateTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (DEFAULT_TABLE table=default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88 [id=2c1e80f117244d4a91c59578fbc1245f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.319382 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235faeb3c35d46d2882ffcd8cb63ddb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.319459 1178 ts_tablet_manager.cc:1428] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250112 20:51:32.319712 1178 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.321336 1163 tablet_bootstrap.cc:492] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.321439 1178 tablet_bootstrap.cc:654] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.321473 1163 ts_tablet_manager.cc:1397] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.000s
I20250112 20:51:32.322127 1163 raft_consensus.cc:357] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.322233 1186 ts_tablet_manager.cc:1939] T b32dc02497f64a2e9ad99da6c9115d5d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:32.322309 1163 raft_consensus.cc:383] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.322362 1163 raft_consensus.cc:738] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.322595 1163 consensus_queue.cc:260] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.296386 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4561956c97824824a55283a8da21929e (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.323361 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4561956c97824824a55283a8da21929e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.324020 1163 ts_tablet_manager.cc:1428] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:32.324275 1163 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.324808 1186 tablet_replica.cc:331] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.325055 1186 raft_consensus.cc:2238] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.325224 1186 raft_consensus.cc:2267] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.325706 1186 ts_tablet_manager.cc:1905] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.298450 755 tablet_service.cc:1467] Processing CreateTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (DEFAULT_TABLE table=default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.298864 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9856e6b625d04f17b0869cee08f69142 (DEFAULT_TABLE table=default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.329232 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f6a1e9804e6ab822f9d4c004b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.332989 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9856e6b625d04f17b0869cee08f69142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.335744 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_458c1d4d727849b88b7502de13ea9a40 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 2 got 0
I20250112 20:51:32.339988 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 (DEFAULT_TABLE table=default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.340144 1163 tablet_bootstrap.cc:654] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.340397 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3343e1d7f1d04a498bd0ee00311b6a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.289499 1185 ts_tablet_manager.cc:1918] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.340752 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 (DEFAULT_TABLE table=default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.340878 1185 log.cc:1198] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/0e2235e2a63c4e1192b3260085908445
I20250112 20:51:32.341459 1185 ts_tablet_manager.cc:1939] T 0e2235e2a63c4e1192b3260085908445 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.342568 1185 tablet_replica.cc:331] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.342926 1185 raft_consensus.cc:2238] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.343190 1185 raft_consensus.cc:2267] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.343869 1185 ts_tablet_manager.cc:1905] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.345058 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3343e1d7f1d04a498bd0ee00311b6a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.346513 1163 tablet_bootstrap.cc:492] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.346650 1163 ts_tablet_manager.cc:1397] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250112 20:51:32.347656 1163 raft_consensus.cc:357] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.347862 1163 raft_consensus.cc:383] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.347944 1163 raft_consensus.cc:738] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.347851 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 (DEFAULT_TABLE table=default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.348140 1163 consensus_queue.cc:260] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.348711 1163 ts_tablet_manager.cc:1428] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:32.348949 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3343e1d7f1d04a498bd0ee00311b6a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.349400 1164 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:32.349589 1164 ts_tablet_manager.cc:1397] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.145s user 0.001s sys 0.004s
I20250112 20:51:32.350445 1164 raft_consensus.cc:357] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.350724 1164 raft_consensus.cc:383] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.350682 1663 consensus_queue.cc:1035] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.350842 1163 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.352524 1163 tablet_bootstrap.cc:654] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.352739 1704 raft_consensus.cc:491] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.352945 1704 raft_consensus.cc:513] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.353061 1185 ts_tablet_manager.cc:1918] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.353590 1704 leader_election.cc:290] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.354346 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.354530 1047 raft_consensus.cc:2463] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:32.355036 847 leader_election.cc:304] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:32.355558 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.355770 775 raft_consensus.cc:2463] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:32.356410 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d0018e1eabf24241a358a7e00b6b5907 (DEFAULT_TABLE table=default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.356715 749 tablet_service.cc:1467] Processing CreateTablet for tablet d0018e1eabf24241a358a7e00b6b5907 (DEFAULT_TABLE table=default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.356742 881 tablet_service.cc:1467] Processing CreateTablet for tablet d0018e1eabf24241a358a7e00b6b5907 (DEFAULT_TABLE table=default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.357023 1028 tablet_service.cc:1467] Processing CreateTablet for tablet d7117edf078848c0a1c83ab8e804dfed (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.357123 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0018e1eabf24241a358a7e00b6b5907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.357120 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0018e1eabf24241a358a7e00b6b5907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.357415 753 tablet_service.cc:1467] Processing CreateTablet for tablet d7117edf078848c0a1c83ab8e804dfed (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.357550 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7117edf078848c0a1c83ab8e804dfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.357762 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7117edf078848c0a1c83ab8e804dfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.361115 1704 raft_consensus.cc:2798] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:32.361200 1704 raft_consensus.cc:491] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:32.361248 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0018e1eabf24241a358a7e00b6b5907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.361287 1704 raft_consensus.cc:3054] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.363583 1704 raft_consensus.cc:513] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.364248 1704 leader_election.cc:290] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.365737 1164 raft_consensus.cc:738] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.366233 1164 consensus_queue.cc:260] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.366756 877 tablet_service.cc:1467] Processing CreateTablet for tablet d7117edf078848c0a1c83ab8e804dfed (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.367108 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7117edf078848c0a1c83ab8e804dfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.369199 1164 ts_tablet_manager.cc:1428] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250112 20:51:32.369709 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.369971 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.369554 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.370414 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6b4882a3d744bda5677aa9e01a53f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.370790 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.371409 1185 log.cc:1198] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:32.371511 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:32.371714 1047 raft_consensus.cc:3054] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.371778 1185 ts_tablet_manager.cc:1939] T bc5af0bc83bc415b972cbbe0fb39f75f P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.372799 1186 ts_tablet_manager.cc:1918] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.372920 1186 log.cc:1198] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/6f4421d16e3f423794c44970a2469b10
I20250112 20:51:32.373165 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.373258 1186 ts_tablet_manager.cc:1939] T 6f4421d16e3f423794c44970a2469b10 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:32.373575 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d3f3e597f4876bce5852961f37ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.373906 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet b32dc02497f64a2e9ad99da6c9115d5d with delete_type TABLET_DATA_DELETED (replaced by tablet 81f0875e271d4aaf9dc5da6aa8113ed0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.374119 1186 tablet_replica.cc:331] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.374277 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.374341 1186 raft_consensus.cc:2238] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.374402 1047 raft_consensus.cc:2463] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.374625 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b414be57990432bb786daf5faf0cc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.374704 1186 raft_consensus.cc:2267] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.374979 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f39195945024f4fba9942faec70bc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:25 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.375365 1185 tablet_replica.cc:331] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.375366 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet ef89229192934339954566e366b6891c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.375561 1185 raft_consensus.cc:2238] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.375787 1185 pending_rounds.cc:62] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:32.375783 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b70ffb3e3341d398d36ff76993e9ac with delete_type TABLET_DATA_DELETED (replaced by tablet d770e0399e80491f8de9b08d1eb2f139) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.375865 1185 pending_rounds.cc:69] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585014866661376 op_type: NO_OP noop_request { }
I20250112 20:51:32.375982 1185 raft_consensus.cc:2883] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:32.376080 1185 raft_consensus.cc:2267] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.376129 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2afa5b0717df4f83a2b44efac12fc06b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.376542 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet cc971026f6b44db6af8215890ba71075 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.377038 1184 tablet_replica.cc:331] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:32.377110 1185 ts_tablet_manager.cc:1905] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.377306 1184 raft_consensus.cc:2238] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.377454 1184 raft_consensus.cc:2267] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.377779 1186 ts_tablet_manager.cc:1905] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.377813 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c38c8594a384689ae57c26174765284" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:32.377950 1184 ts_tablet_manager.cc:1905] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.377993 775 raft_consensus.cc:3054] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.379573 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8056e2e94fa94fc0b7f03fede018dd8e (DEFAULT_TABLE table=default.Table_8db2ac83bd80420da5eed5abfe3c3fda [id=8c792b622e26451f99361963d1ea0ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.379954 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8056e2e94fa94fc0b7f03fede018dd8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.381085 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.381346 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.381464 847 leader_election.cc:304] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:32.381520 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4e304a1212412f8e99bfd222a110b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.381978 1026 tablet_service.cc:1467] Processing CreateTablet for tablet db9178cf782d4bd28eba90cc03796885 (DEFAULT_TABLE table=default.Table_986d0d39b1fa41abb2166e064df20564 [id=16c454ac411b44a3b5d282369bb13e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.382221 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.382355 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9178cf782d4bd28eba90cc03796885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.382602 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6b4882a3d744bda5677aa9e01a53f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.382764 1790 raft_consensus.cc:2798] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.382917 1790 raft_consensus.cc:695] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:32.383219 1790 consensus_queue.cc:237] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.384366 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.384573 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.384815 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.384933 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.385042 878 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4e304a1212412f8e99bfd222a110b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.387149 876 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:32.387310 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 54ab0ae10f09469fa4f913e049002481 because the tablet was not found. No further retry: Not found: Tablet not found: 54ab0ae10f09469fa4f913e049002481
I20250112 20:51:32.387504 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:32.387531 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6092a31de9bc47e4bae5f5aebb180c6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092a31de9bc47e4bae5f5aebb180c6d already in progress: creating tablet
W20250112 20:51:32.387820 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 48e390eb2e404250a8b6edba2ee4f6d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e390eb2e404250a8b6edba2ee4f6d6 already in progress: creating tablet
I20250112 20:51:32.388010 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]) successfully deleted
I20250112 20:51:32.379170 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8056e2e94fa94fc0b7f03fede018dd8e (DEFAULT_TABLE table=default.Table_8db2ac83bd80420da5eed5abfe3c3fda [id=8c792b622e26451f99361963d1ea0ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.388324 876 tablet_service.cc:1467] Processing CreateTablet for tablet db9178cf782d4bd28eba90cc03796885 (DEFAULT_TABLE table=default.Table_986d0d39b1fa41abb2166e064df20564 [id=16c454ac411b44a3b5d282369bb13e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.388602 878 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.388649 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d477fb5e5ad5447bb182bc9998561a43 (DEFAULT_TABLE table=default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.389022 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.389086 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d477fb5e5ad5447bb182bc9998561a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.389446 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 (DEFAULT_TABLE table=default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.389730 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.389791 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f669a739d94f99a5b7e9ccccd29545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.388645 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9178cf782d4bd28eba90cc03796885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.389087 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 921ae136d5e2438e950b96f156f23da3 (DEFAULT_TABLE table=default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.391795 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921ae136d5e2438e950b96f156f23da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.392915 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8056e2e94fa94fc0b7f03fede018dd8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.393077 878 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccb593413124921af94d0b855789b52 (DEFAULT_TABLE table=default.Table_c37a19db737147b3a05d4cd80380f59e [id=55f399c9f7d34345a15b18ed312bf623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.393363 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccb593413124921af94d0b855789b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.396224 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 226ea7f38f1445c598676715a3036227 (DEFAULT_TABLE table=default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.396569 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 226ea7f38f1445c598676715a3036227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.388649 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d (DEFAULT_TABLE table=default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.401479 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.401594 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4e304a1212412f8e99bfd222a110b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.401578 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d29316b9d2441bca97f56bdb0e56f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.401974 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.402066 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccb593413124921af94d0b855789b52 (DEFAULT_TABLE table=default.Table_c37a19db737147b3a05d4cd80380f59e [id=55f399c9f7d34345a15b18ed312bf623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.402408 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6b4882a3d744bda5677aa9e01a53f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.402609 746 tablet_service.cc:1467] Processing CreateTablet for tablet 921ae136d5e2438e950b96f156f23da3 (DEFAULT_TABLE table=default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.402631 750 tablet_service.cc:1467] Processing CreateTablet for tablet db9178cf782d4bd28eba90cc03796885 (DEFAULT_TABLE table=default.Table_986d0d39b1fa41abb2166e064df20564 [id=16c454ac411b44a3b5d282369bb13e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.403056 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921ae136d5e2438e950b96f156f23da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.403102 751 tablet_service.cc:1467] Processing CreateTablet for tablet d477fb5e5ad5447bb182bc9998561a43 (DEFAULT_TABLE table=default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.403409 1701 raft_consensus.cc:491] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.403558 1701 raft_consensus.cc:513] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.404160 1701 leader_election.cc:290] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.404554 748 tablet_service.cc:1467] Processing CreateTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 (DEFAULT_TABLE table=default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.405130 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d (DEFAULT_TABLE table=default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.405278 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.405541 740 tablet_service.cc:1467] Processing CreateTablet for tablet f143227e79844a7393b323222bf7eaef (DEFAULT_TABLE table=default.Table_65e8cf7dc1134afaae922506cf15579b [id=467b97b4e4694b2f8551e237429bd0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.405903 745 tablet_service.cc:1467] Processing CreateTablet for tablet 226ea7f38f1445c598676715a3036227 (DEFAULT_TABLE table=default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.406286 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d477fb5e5ad5447bb182bc9998561a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.409013 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 226ea7f38f1445c598676715a3036227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.409744 1178 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.409931 1178 ts_tablet_manager.cc:1397] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.004s
I20250112 20:51:32.410806 1178 raft_consensus.cc:357] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.411026 1178 raft_consensus.cc:383] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.411098 1178 raft_consensus.cc:738] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.411479 1178 consensus_queue.cc:260] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.380735 775 raft_consensus.cc:2463] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.412235 1178 ts_tablet_manager.cc:1428] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:32.412570 1178 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.412994 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f669a739d94f99a5b7e9ccccd29545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.414424 1178 tablet_bootstrap.cc:654] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.416075 882 tablet_service.cc:1467] Processing CreateTablet for tablet 921ae136d5e2438e950b96f156f23da3 (DEFAULT_TABLE table=default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.416160 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b32dc02497f64a2e9ad99da6c9115d5d (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:32.416296 884 tablet_service.cc:1467] Processing CreateTablet for tablet f143227e79844a7393b323222bf7eaef (DEFAULT_TABLE table=default.Table_65e8cf7dc1134afaae922506cf15579b [id=467b97b4e4694b2f8551e237429bd0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.416424 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921ae136d5e2438e950b96f156f23da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.416455 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 6f4421d16e3f423794c44970a2469b10 (table default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]) successfully deleted
I20250112 20:51:32.416581 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f143227e79844a7393b323222bf7eaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.417274 890 tablet_service.cc:1467] Processing CreateTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 (DEFAULT_TABLE table=default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.417742 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f669a739d94f99a5b7e9ccccd29545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.418409 880 tablet_service.cc:1467] Processing CreateTablet for tablet 226ea7f38f1445c598676715a3036227 (DEFAULT_TABLE table=default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.418745 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 226ea7f38f1445c598676715a3036227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.420058 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d29316b9d2441bca97f56bdb0e56f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.420640 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f143227e79844a7393b323222bf7eaef (DEFAULT_TABLE table=default.Table_65e8cf7dc1134afaae922506cf15579b [id=467b97b4e4694b2f8551e237429bd0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.421125 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f143227e79844a7393b323222bf7eaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.422204 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccb593413124921af94d0b855789b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.422365 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.427618 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0db67ae601df4b229f938deaeb0c6f8c" } new_table_name: "default.Table_fe56f26bf2934585834bfaf76f5a1d08" modify_external_catalogs: true
W20250112 20:51:32.428012 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 48e390eb2e404250a8b6edba2ee4f6d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e390eb2e404250a8b6edba2ee4f6d6 already in progress: creating tablet
I20250112 20:51:32.430518 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111),
I20250112 20:51:32.430682 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 20:51:32.432451 1720 raft_consensus.cc:491] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.432618 1720 raft_consensus.cc:513] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.432907 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]) successfully deleted
I20250112 20:51:32.433364 1720 leader_election.cc:290] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.434186 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.434136 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 (DEFAULT_TABLE table=default.Table_23317c504a8b4a7392676291f5385b74 [id=b3b49691fa8a41dda2abad8d2f59e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.434492 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f9b46da06a476ab9f2a93ca74ff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.435272 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8056e2e94fa94fc0b7f03fede018dd8e (DEFAULT_TABLE table=default.Table_8db2ac83bd80420da5eed5abfe3c3fda [id=8c792b622e26451f99361963d1ea0ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.435616 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8056e2e94fa94fc0b7f03fede018dd8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.441035 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.441331 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.441681 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccb593413124921af94d0b855789b52 (DEFAULT_TABLE table=default.Table_c37a19db737147b3a05d4cd80380f59e [id=55f399c9f7d34345a15b18ed312bf623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.442018 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccb593413124921af94d0b855789b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.443713 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.447067 879 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.448101 889 tablet_service.cc:1467] Processing CreateTablet for tablet d477fb5e5ad5447bb182bc9998561a43 (DEFAULT_TABLE table=default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.448510 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.449441 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.449406 888 tablet_service.cc:1467] Processing CreateTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 (DEFAULT_TABLE table=default.Table_23317c504a8b4a7392676291f5385b74 [id=b3b49691fa8a41dda2abad8d2f59e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.450250 1186 ts_tablet_manager.cc:1918] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.449575 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86d0ef63d94a55adf79bb136ea89e8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.451865 743 tablet_service.cc:1467] Processing CreateTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 (DEFAULT_TABLE table=default.Table_23317c504a8b4a7392676291f5385b74 [id=b3b49691fa8a41dda2abad8d2f59e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.452847 1164 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:32.452841 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1a04c223889f424e831e82a52a84f804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a04c223889f424e831e82a52a84f804 already in progress: creating tablet
W20250112 20:51:32.453368 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d0018e1eabf24241a358a7e00b6b5907 because the tablet was not found. No further retry: Not found: Tablet not found: d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:32.454825 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.454922 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d477fb5e5ad5447bb182bc9998561a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.454659 1186 log.cc:1198] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7f39195945024f4fba9942faec70bc31
I20250112 20:51:32.456017 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f143227e79844a7393b323222bf7eaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.456878 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9178cf782d4bd28eba90cc03796885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.457376 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f9b46da06a476ab9f2a93ca74ff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.459231 878 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d (DEFAULT_TABLE table=default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.460237 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:32.460239 710 consensus_peers.cc:487] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f4421d16e3f423794c44970a2469b10. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:32.463625 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.464138 910 raft_consensus.cc:2388] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:32.464825 983 leader_election.cc:304] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.465863 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.466085 775 raft_consensus.cc:2463] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:32.467356 1720 raft_consensus.cc:3054] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.462845 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d770e0399e80491f8de9b08d1eb2f139 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
W20250112 20:51:32.468755 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bc05d287d81c46558acdd9a42edda6f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc05d287d81c46558acdd9a42edda6f8 already in progress: creating tablet
I20250112 20:51:32.469233 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4c3f359655c94f1d905943069548845b (table default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]) successfully deleted
W20250112 20:51:32.469944 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 54ab0ae10f09469fa4f913e049002481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54ab0ae10f09469fa4f913e049002481 already in progress: creating tablet
I20250112 20:51:32.471977 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 0e2235e2a63c4e1192b3260085908445 (table default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]) successfully deleted
I20250112 20:51:32.473133 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f9b46da06a476ab9f2a93ca74ff736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.474596 1184 ts_tablet_manager.cc:1918] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.482200 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461]). Expected version 1 got 0
I20250112 20:51:32.482511 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_458c1d4d727849b88b7502de13ea9a40 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 2 got 0
I20250112 20:51:32.482751 451 catalog_manager.cc:5526] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:32.483325 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:32.483608 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:32.483796 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:32.484033 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 24c8dc85ac9240598d6e413deac19abe (table default.Table_b2ca205a6c6f470c896d5194717dbf72 [id=3a4de1859ffd4db3931ce68febdd8836]). Expected version 2 got 0
W20250112 20:51:32.484354 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba86d0ef63d94a55adf79bb136ea89e8 consensus state has no leader
I20250112 20:51:32.485265 1720 raft_consensus.cc:2743] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.485543 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:45153
--webserver_interface=127.0.104.60
--webserver_port=44167
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 20:51:32.492198 1184 log.cc:1198] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b13d3f3e597f4876bce5852961f37ea0
I20250112 20:51:32.492980 1184 ts_tablet_manager.cc:1939] T b13d3f3e597f4876bce5852961f37ea0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:32.503501 1163 tablet_bootstrap.cc:492] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:32.504637 847 leader_election.cc:343] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.506740 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d29316b9d2441bca97f56bdb0e56f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.507270 1178 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.507447 1178 ts_tablet_manager.cc:1397] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.000s
I20250112 20:51:32.508459 1178 raft_consensus.cc:357] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.508865 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet bc5af0bc83bc415b972cbbe0fb39f75f (table default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]) successfully deleted
W20250112 20:51:32.510102 440 catalog_manager.cc:4945] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): alter failed for tablet fc63526f932841509277f4726303c801 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]),no further retry: Not found: Tablet not found: fc63526f932841509277f4726303c801
I20250112 20:51:32.512859 1587 raft_consensus.cc:491] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.509719 1178 raft_consensus.cc:383] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.513965 1186 ts_tablet_manager.cc:1939] T 7f39195945024f4fba9942faec70bc31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:32.518311 1185 ts_tablet_manager.cc:1918] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.518451 1185 log.cc:1198] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1c7a54ce78854137ab2d12e9ce055565
I20250112 20:51:32.518833 1185 ts_tablet_manager.cc:1939] T 1c7a54ce78854137ab2d12e9ce055565 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.521232 1164 tablet_bootstrap.cc:654] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.523159 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_458c1d4d727849b88b7502de13ea9a40 [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 2 got 0
I20250112 20:51:32.523370 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:32.523504 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:32.523651 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:32.523850 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:32.520731 1587 raft_consensus.cc:513] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.524099 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]). Expected version 1 got 0
I20250112 20:51:32.524267 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:32.524515 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c2a8f71233d04ca39e49932308e56492 (table default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6]). Expected version 3 got 0
I20250112 20:51:32.525329 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b13d3f3e597f4876bce5852961f37ea0 (table default.Table_2feab5fd4f5b4753ab157456f04f7e70 [id=bfb9b545f79c4f5ab615d0c0c928f5b8]) successfully deleted
I20250112 20:51:32.526669 1587 leader_election.cc:290] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.526979 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]). Expected version 1 got 0
I20250112 20:51:32.527140 1178 raft_consensus.cc:738] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.527663 1185 tablet_replica.cc:331] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.528283 1185 raft_consensus.cc:2238] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.528443 1185 raft_consensus.cc:2267] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.527870 1178 consensus_queue.cc:260] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.527158 1186 tablet_replica.cc:331] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.531924 742 tablet_service.cc:1467] Processing CreateTablet for tablet e72a6c8e18934b789a048d5aaded121b (DEFAULT_TABLE table=default.Table_284c5bfc11f1458dbade76ac78b87f20 [id=f83f83db15594cd9928fccf547cad7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.532513 754 tablet_service.cc:1467] Processing CreateTablet for tablet 56544c3b0cf140089baa678765a2eb79 (DEFAULT_TABLE table=default.Table_58f5707fbcdd4c339243ab9c0645d2aa [id=462ba020c7414465b636c07442083e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.534406 1186 raft_consensus.cc:2238] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.534579 1186 raft_consensus.cc:2267] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.534641 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0093059b34fb40609289f578978c8750 (DEFAULT_TABLE table=default.Table_07ca89e09c4f4d28b8d2ab35a14a3d22 [id=aeb164cfc97d40d0b4dc6c400ab6a35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.536398 453 catalog_manager.cc:5526] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:32.537056 756 tablet_service.cc:1467] Processing CreateTablet for tablet 161951bfabff485dbf7dceeac2d4128d (DEFAULT_TABLE table=default.Table_bfe92375d9a04e5aa7fc64840b0bc050 [id=60b6e28026a74197a4ba790310e1680d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.537577 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]). Expected version 1 got 0
I20250112 20:51:32.537987 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:32.538184 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:32.540117 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]). Expected version 1 got 0
I20250112 20:51:32.541019 753 tablet_service.cc:1467] Processing CreateTablet for tablet b930069b6f1b49439dbc0a7abd3fba72 (DEFAULT_TABLE table=default.Table_29c0f892c9954a598f7f27c4be906a91 [id=61af9ec286544b919c9f053b647bc6fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.543565 744 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f3389061143509354f37ae93ef0c7 (DEFAULT_TABLE table=default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.543972 739 tablet_service.cc:1467] Processing CreateTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (DEFAULT_TABLE table=default.Table_b2428906915d47bd9fb06b88e971194c [id=fb7c117897c24636b7e2dd8109e782d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.544428 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 (DEFAULT_TABLE table=default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.545055 747 tablet_service.cc:1467] Processing CreateTablet for tablet 53551931b23d43da89e9193cdf034a63 (DEFAULT_TABLE table=default.Table_c5469ad8d235490ebfbbb933c659d6ec [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.547883 752 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.548985 1178 ts_tablet_manager.cc:1428] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250112 20:51:32.549337 1178 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.549547 1790 raft_consensus.cc:491] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.550657 1185 ts_tablet_manager.cc:1905] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.551541 1790 raft_consensus.cc:513] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:32.541327 746 tablet_service.cc:1467] Processing CreateTablet for tablet 577dc789f0cb48f49168124b37a6fba0 (DEFAULT_TABLE table=default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.554126 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a6c8e18934b789a048d5aaded121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.555446 1163 ts_tablet_manager.cc:1397] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.003s
I20250112 20:51:32.556464 1163 raft_consensus.cc:357] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.556735 1163 raft_consensus.cc:383] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.556842 1163 raft_consensus.cc:738] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.557155 1163 consensus_queue.cc:260] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.558574 1186 ts_tablet_manager.cc:1905] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.559327 1790 leader_election.cc:290] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:32.563505 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.564319 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0093059b34fb40609289f578978c8750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.564369 745 tablet_service.cc:1467] Processing CreateTablet for tablet cbc22207e9204b73877c74f61ebba8a8 (DEFAULT_TABLE table=default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.566255 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d97ac93467a4a7a8ae44d52719f191c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.566484 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2f3389061143509354f37ae93ef0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.567370 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ef01b41b6140d6bdad07bc79dd7dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.567696 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b930069b6f1b49439dbc0a7abd3fba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.567930 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd47d1c1a4b42bdbff97f619dc95134. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.573714 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a6951a044a6446d9b2a2bdd8d43f305b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.574301 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6951a044a6446d9b2a2bdd8d43f305b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.574900 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_21b7926d70e24852a7e66dbb6449e4ae" } new_table_name: "default.Table_41daf866320a48988a8961ef716b8cf0" modify_external_catalogs: true
I20250112 20:51:32.575001 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_22041bd544d842dd8bba1f206c270a4e" } modify_external_catalogs: true
I20250112 20:51:32.575078 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_56e32a726bf14d0193beb81aef87a2de" } modify_external_catalogs: true
I20250112 20:51:32.575220 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_22041bd544d842dd8bba1f206c270a4e" } modify_external_catalogs: true
I20250112 20:51:32.575395 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_56e32a726bf14d0193beb81aef87a2de" } modify_external_catalogs: true
I20250112 20:51:32.575939 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4283dadebf89422795d890e3991db698"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:32.578008 1720 consensus_queue.cc:1035] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20250112 20:51:32.578953 1720 raft_consensus.cc:491] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.579113 1720 raft_consensus.cc:513] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.579802 1720 leader_election.cc:290] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.580319 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 577dc789f0cb48f49168124b37a6fba0 (DEFAULT_TABLE table=default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.580827 1184 tablet_replica.cc:331] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:32.581542 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577dc789f0cb48f49168124b37a6fba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.583503 1638 raft_consensus.cc:491] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:32.588279 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 267ebb3c30194d9fa6a092d9bec4eadf consensus state has no leader
I20250112 20:51:32.588245 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43f4ea15650c4619afdabdb3c169df86" } modify_external_catalogs: true
I20250112 20:51:32.588435 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43f4ea15650c4619afdabdb3c169df86" } modify_external_catalogs: true
I20250112 20:51:32.589006 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aa5b2f32e08b4d1d8c28904dbda61f18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.589643 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa5b2f32e08b4d1d8c28904dbda61f18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.589882 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0cbb400711764b7a8fffe933e3334d42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.590390 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cbb400711764b7a8fffe933e3334d42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.590715 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b3533fddf0f8403bb052fb6b07ab0b20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.591176 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3533fddf0f8403bb052fb6b07ab0b20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.591470 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_31360e3161b04902a56502dbe4608f86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.592111 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31360e3161b04902a56502dbe4608f86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.590355 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]). Expected version 3 got 0
I20250112 20:51:32.593269 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]). Expected version 1 got 0
I20250112 20:51:32.595315 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:32.595917 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:32.600052 1638 raft_consensus.cc:513] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.600827 738 tablet_service.cc:1467] Processing CreateTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.602429 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53551931b23d43da89e9193cdf034a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.603071 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56544c3b0cf140089baa678765a2eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.603932 1638 leader_election.cc:290] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:32.602959 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4283dadebf89422795d890e3991db698 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.604100 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:32.605341 710 leader_election.cc:343] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.606017 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577dc789f0cb48f49168124b37a6fba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.606599 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229e4ef771b4032a4bfae3190a851e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.607059 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161951bfabff485dbf7dceeac2d4128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.608343 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5b10576bd34af5b02f8d8dcc3759ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.608592 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbc22207e9204b73877c74f61ebba8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.613883 1164 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:32.614099 1164 ts_tablet_manager.cc:1397] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.004s
I20250112 20:51:32.615018 1164 raft_consensus.cc:357] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.615248 1164 raft_consensus.cc:383] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.615326 1164 raft_consensus.cc:738] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.615384 1186 ts_tablet_manager.cc:1918] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.615478 751 tablet_service.cc:1467] Processing CreateTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 (DEFAULT_TABLE table=default.Table_6c73da0cad5b49beb8a7ec45996a6d6d [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.615825 1186 log.cc:1198] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/bcbbefa7b55041a6a6b50b3c96c45833
I20250112 20:51:32.616020 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b8cc99742f4f9ba68eb1aa17cab248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.616143 1186 ts_tablet_manager.cc:1939] T bcbbefa7b55041a6a6b50b3c96c45833 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:32.616955 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 110640us
I20250112 20:51:32.617036 1814 raft_consensus.cc:491] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.619781 847 leader_election.cc:304] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:32.619366 1814 raft_consensus.cc:513] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.620185 1790 raft_consensus.cc:2798] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:32.620301 1790 raft_consensus.cc:491] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:32.620380 1790 raft_consensus.cc:3054] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.623225 1587 raft_consensus.cc:491] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.623400 1587 raft_consensus.cc:513] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.623091 1790 raft_consensus.cc:513] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.624059 1587 leader_election.cc:290] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.624399 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.624594 1814 leader_election.cc:290] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.624606 774 raft_consensus.cc:2463] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:32.625370 1780 tablet.cc:1721] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:32.630333 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.630527 1049 raft_consensus.cc:2463] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:32.631755 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d770e0399e80491f8de9b08d1eb2f139" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.632283 878 tablet_service.cc:1467] Processing CreateTablet for tablet 577dc789f0cb48f49168124b37a6fba0 (DEFAULT_TABLE table=default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.632706 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577dc789f0cb48f49168124b37a6fba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.633539 1761 tablet.cc:1721] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:32.634675 749 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 (DEFAULT_TABLE table=default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.635038 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd34e829d0645ea80daaff5d27cb266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.639851 890 tablet_service.cc:1467] Processing CreateTablet for tablet b930069b6f1b49439dbc0a7abd3fba72 (DEFAULT_TABLE table=default.Table_29c0f892c9954a598f7f27c4be906a91 [id=61af9ec286544b919c9f053b647bc6fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.640210 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b930069b6f1b49439dbc0a7abd3fba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.640383 876 tablet_service.cc:1467] Processing CreateTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.640794 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d97ac93467a4a7a8ae44d52719f191c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.641005 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 (DEFAULT_TABLE table=default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.641434 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd47d1c1a4b42bdbff97f619dc95134. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.641561 847 leader_election.cc:304] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
W20250112 20:51:32.641839 847 leader_election.cc:343] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.642438 878 tablet_service.cc:1467] Processing CreateTablet for tablet cbc22207e9204b73877c74f61ebba8a8 (DEFAULT_TABLE table=default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.642799 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbc22207e9204b73877c74f61ebba8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.644770 882 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f3389061143509354f37ae93ef0c7 (DEFAULT_TABLE table=default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.645079 886 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.640770 877 tablet_service.cc:1467] Processing CreateTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.639870 888 tablet_service.cc:1467] Processing CreateTablet for tablet e72a6c8e18934b789a048d5aaded121b (DEFAULT_TABLE table=default.Table_284c5bfc11f1458dbade76ac78b87f20 [id=f83f83db15594cd9928fccf547cad7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.645615 874 tablet_service.cc:1467] Processing CreateTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 (DEFAULT_TABLE table=default.Table_6c73da0cad5b49beb8a7ec45996a6d6d [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.645160 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2f3389061143509354f37ae93ef0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.646159 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 (DEFAULT_TABLE table=default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.646536 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd34e829d0645ea80daaff5d27cb266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.647331 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229e4ef771b4032a4bfae3190a851e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.647744 773 raft_consensus.cc:1270] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:32.648141 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:32.648335 776 raft_consensus.cc:3054] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.648931 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.649168 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.649412 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b8cc99742f4f9ba68eb1aa17cab248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.651098 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a6c8e18934b789a048d5aaded121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.651654 776 raft_consensus.cc:2463] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.652406 889 tablet_service.cc:1467] Processing CreateTablet for tablet 0093059b34fb40609289f578978c8750 (DEFAULT_TABLE table=default.Table_07ca89e09c4f4d28b8d2ab35a14a3d22 [id=aeb164cfc97d40d0b4dc6c400ab6a35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.655416 884 tablet_service.cc:1467] Processing CreateTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (DEFAULT_TABLE table=default.Table_b2428906915d47bd9fb06b88e971194c [id=fb7c117897c24636b7e2dd8109e782d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.666533 1043 raft_consensus.cc:1270] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:32.667671 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1c7a54ce78854137ab2d12e9ce055565 (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:32.668283 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7f39195945024f4fba9942faec70bc31 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:32.652796 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0093059b34fb40609289f578978c8750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.671502 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5b10576bd34af5b02f8d8dcc3759ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.672837 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3f359655c94f1d905943069548845b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.674836 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ef01b41b6140d6bdad07bc79dd7dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.677660 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0712ab2de43eea8c77ab39a971a2c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:32.679075 743 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.679060 880 tablet_service.cc:1467] Processing CreateTablet for tablet 56544c3b0cf140089baa678765a2eb79 (DEFAULT_TABLE table=default.Table_58f5707fbcdd4c339243ab9c0645d2aa [id=462ba020c7414465b636c07442083e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.679272 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.679428 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56544c3b0cf140089baa678765a2eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.679474 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.679966 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.680167 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.680315 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.624082 1790 leader_election.cc:290] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.682181 1790 raft_consensus.cc:2798] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:32.682261 1790 raft_consensus.cc:491] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:32.682381 1790 raft_consensus.cc:3054] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.683739 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.683974 743 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.695638 881 tablet_service.cc:1467] Processing CreateTablet for tablet 53551931b23d43da89e9193cdf034a63 (DEFAULT_TABLE table=default.Table_c5469ad8d235490ebfbbb933c659d6ec [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.699971 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53551931b23d43da89e9193cdf034a63. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:32.707548 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:45153, user_credentials={real_user=slave}} blocked reactor thread for 70901.5us
I20250112 20:51:32.710180 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:32.718024 1790 raft_consensus.cc:513] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.725306 879 tablet_service.cc:1467] Processing CreateTablet for tablet 161951bfabff485dbf7dceeac2d4128d (DEFAULT_TABLE table=default.Table_bfe92375d9a04e5aa7fc64840b0bc050 [id=60b6e28026a74197a4ba790310e1680d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.725541 1790 leader_election.cc:290] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.725792 1184 raft_consensus.cc:2238] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:32.725870 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161951bfabff485dbf7dceeac2d4128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.726037 1184 raft_consensus.cc:2267] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.726948 1184 ts_tablet_manager.cc:1905] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:32.728446 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 59678.4us
I20250112 20:51:32.730406 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet bcbbefa7b55041a6a6b50b3c96c45833 (table default.Table_69153618434846ad9d90837893bea2da [id=be8442e96b3b4635ba6dac3956ee9de3]) successfully deleted
I20250112 20:51:32.732333 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5b2f736731d74cdfb099e7a498bf92a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:32.732460 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6c73da0cad5b49beb8a7ec45996a6d6d" } new_table_name: "default.Table_25113f809e324283a5b53261123398f1" modify_external_catalogs: true
I20250112 20:51:32.732738 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ae1eb78e32524283a93e97294e33bf5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.733004 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b2f736731d74cdfb099e7a498bf92a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:32.733192 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae1eb78e32524283a93e97294e33bf5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.732483 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ca3d4ace4e204dd799bd05ed211341a0" } new_table_name: "default.Table_9dde7822fdb743bda1e94189dd58b5ad" modify_external_catalogs: true
I20250112 20:51:32.734361 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8fc6da3983cb406e8e10ecf3f3fd239b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.734829 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc6da3983cb406e8e10ecf3f3fd239b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.736142 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b0787900a74e4ae79083035c64b84c00" } modify_external_catalogs: true
I20250112 20:51:32.736364 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b0787900a74e4ae79083035c64b84c00" } modify_external_catalogs: true
I20250112 20:51:32.738401 1186 tablet_replica.cc:331] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.738699 1186 raft_consensus.cc:2238] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.739336 1186 raft_consensus.cc:2267] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.741036 1186 ts_tablet_manager.cc:1905] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.747181 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1b743410da4942a091143beef17273fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:32.747728 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b743410da4942a091143beef17273fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:32.751930 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_458c1d4d727849b88b7502de13ea9a40" } new_table_name: "default.Table_c275eb298fde46e7b70911f21c75fffe" modify_external_catalogs: true
I20250112 20:51:32.762701 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:32.763252 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49b8cc99742f4f9ba68eb1aa17cab248 consensus state has no leader
W20250112 20:51:32.763833 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0daaeb0bf6046da8de969b1f4c71c84 consensus state has no leader
I20250112 20:51:32.764101 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 7deb6e37f75e4aebacb04c4553049940 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:32.764855 447 master_service.cc:946] ReplaceTablet: received request to replace tablet d0018e1eabf24241a358a7e00b6b5907 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:32.766283 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_46fa14b9681540c5a4649563f92f7a53" } modify_external_catalogs: true
I20250112 20:51:32.766475 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_46fa14b9681540c5a4649563f92f7a53" } modify_external_catalogs: true
I20250112 20:51:32.768903 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3b87490cb2634435811d01d9ac40c7ea" } new_table_name: "default.Table_a77dee800e62415db3cada3902c1efcc" modify_external_catalogs: true
I20250112 20:51:32.769629 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 44f11304909b416aa1e9db1abf633a89
I20250112 20:51:32.725908 1790 raft_consensus.cc:491] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.774232 1790 raft_consensus.cc:513] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.775218 1790 leader_election.cc:290] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.776425 1186 ts_tablet_manager.cc:1918] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.776741 1186 log.cc:1198] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/158cfe5ba52c4771af97841885b29f9d
I20250112 20:51:32.777426 1186 ts_tablet_manager.cc:1939] T 158cfe5ba52c4771af97841885b29f9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:32.778038 1186 tablet_replica.cc:331] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:32.782099 910 raft_consensus.cc:1270] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Refusing update from remote peer aefe1db8b19244d891b6bcd8971e9b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:32.783041 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 161951bfabff485dbf7dceeac2d4128d (DEFAULT_TABLE table=default.Table_bfe92375d9a04e5aa7fc64840b0bc050 [id=60b6e28026a74197a4ba790310e1680d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.783603 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161951bfabff485dbf7dceeac2d4128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.784103 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cbc22207e9204b73877c74f61ebba8a8 (DEFAULT_TABLE table=default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.784711 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbc22207e9204b73877c74f61ebba8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.791491 1178 tablet_bootstrap.cc:654] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.793442 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0093059b34fb40609289f578978c8750 (DEFAULT_TABLE table=default.Table_07ca89e09c4f4d28b8d2ab35a14a3d22 [id=aeb164cfc97d40d0b4dc6c400ab6a35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.793715 1163 ts_tablet_manager.cc:1428] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.238s user 0.005s sys 0.006s
I20250112 20:51:32.793982 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0093059b34fb40609289f578978c8750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.794219 1163 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.796171 1163 tablet_bootstrap.cc:654] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.797824 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.799006 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.799544 913 raft_consensus.cc:2388] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:32.801851 1832 raft_consensus.cc:491] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.802065 1832 raft_consensus.cc:513] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.802930 1825 tablet.cc:1721] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:32.807628 1663 consensus_queue.cc:1035] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 20:51:32.812572 1832 leader_election.cc:290] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.816707 452 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 24c8dc85ac9240598d6e413deac19abe (table default.Table_b2ca205a6c6f470c896d5194717dbf72 [id=3a4de1859ffd4db3931ce68febdd8836]). Expected version 2 got 0
I20250112 20:51:32.797288 1824 raft_consensus.cc:491] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.821527 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f11304909b416aa1e9db1abf633a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.821764 1824 raft_consensus.cc:513] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.822647 1824 leader_election.cc:290] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.797469 1821 raft_consensus.cc:491] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.822954 1821 raft_consensus.cc:513] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.823745 1821 leader_election.cc:290] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:32.807008 983 consensus_peers.cc:487] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcbbefa7b55041a6a6b50b3c96c45833. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:32.824628 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7a54ce78854137ab2d12e9ce055565" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.824954 910 raft_consensus.cc:2388] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:32.826099 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.826316 910 raft_consensus.cc:2463] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:32.827011 983 leader_election.cc:304] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:32.827600 1820 raft_consensus.cc:2798] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:32.827710 1820 raft_consensus.cc:491] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:32.827800 1820 raft_consensus.cc:3054] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.829183 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d3669355bc4b6bbc665edaa80c7a9e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.829602 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7deb6e37f75e4aebacb04c4553049940
I20250112 20:51:32.829602 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab5ecd2698b4952a892135b3593106a" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.830075 455 catalog_manager.cc:6596] ReplaceTablet: tablet 7deb6e37f75e4aebacb04c4553049940 deleted and replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f
I20250112 20:51:32.830453 1820 raft_consensus.cc:513] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.830945 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8f636df32bd34438bb54989736eaa431
I20250112 20:51:32.831050 1820 leader_election.cc:290] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:32.833777 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:32.834064 910 raft_consensus.cc:3054] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:32.837215 983 leader_election.cc:343] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:32.837440 983 leader_election.cc:304] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.838775 910 raft_consensus.cc:2463] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:32.839257 1820 raft_consensus.cc:2743] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.839847 983 leader_election.cc:304] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.840231 1820 raft_consensus.cc:3054] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.843232 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.843858 1820 raft_consensus.cc:2743] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.843962 774 raft_consensus.cc:2463] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:32.844532 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:32.844547 983 leader_election.cc:304] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
W20250112 20:51:32.844960 983 leader_election.cc:343] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 1c7a54ce78854137ab2d12e9ce055565
I20250112 20:51:32.845172 983 leader_election.cc:304] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.845891 774 raft_consensus.cc:3054] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.845927 1821 raft_consensus.cc:2798] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.846109 1821 raft_consensus.cc:695] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:32.846457 1821 consensus_queue.cc:237] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.848028 1820 raft_consensus.cc:3054] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.849881 774 raft_consensus.cc:2463] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:32.851529 1820 raft_consensus.cc:2743] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.852968 1178 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.853283 1178 ts_tablet_manager.cc:1397] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.304s user 0.001s sys 0.006s
I20250112 20:51:32.854413 1178 raft_consensus.cc:357] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.854810 1178 raft_consensus.cc:383] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.854960 1178 raft_consensus.cc:738] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.855347 1178 consensus_queue.cc:260] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.807292 1185 ts_tablet_manager.cc:1918] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:32.856030 1178 ts_tablet_manager.cc:1428] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:32.858794 1178 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.860924 1824 raft_consensus.cc:491] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.861240 1824 raft_consensus.cc:513] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.862181 1824 leader_election.cc:290] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:32.863456 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 49b8cc99742f4f9ba68eb1aa17cab248 (table default.Table_25113f809e324283a5b53261123398f1 [id=e3b1932f0a7c43fbb5a8ae879c51b0e6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49b8cc99742f4f9ba68eb1aa17cab248 consensus state has no leader
I20250112 20:51:32.865473 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2235e2a63c4e1192b3260085908445" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.865844 910 raft_consensus.cc:2388] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
W20250112 20:51:32.867533 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 225474us
I20250112 20:51:32.867869 1163 tablet_bootstrap.cc:492] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:32.868080 1163 ts_tablet_manager.cc:1397] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20250112 20:51:32.868613 1825 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.869037 1817 consensus_queue.cc:1035] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.869004 1163 raft_consensus.cc:357] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.869179 1163 raft_consensus.cc:383] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.869328 1163 raft_consensus.cc:738] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:32.869699 847 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.869626 1163 consensus_queue.cc:260] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:32.870028 847 leader_election.cc:343] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.870304 1163 ts_tablet_manager.cc:1428] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 20:51:32.871343 847 leader_election.cc:343] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:32.871627 847 leader_election.cc:304] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.871904 1817 raft_consensus.cc:2743] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.873139 1817 consensus_queue.cc:1035] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.873570 1825 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:32.873775 847 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4c3f359655c94f1d905943069548845b
I20250112 20:51:32.873979 847 leader_election.cc:304] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.874320 1817 raft_consensus.cc:2743] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.875118 847 leader_election.cc:304] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:32.875422 1817 raft_consensus.cc:2798] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.875581 1817 raft_consensus.cc:695] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:32.875885 1817 consensus_queue.cc:237] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:32.884439 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8b414be57990432bb786daf5faf0cc2e because the tablet was not found. No further retry: Not found: Tablet not found: 8b414be57990432bb786daf5faf0cc2e
I20250112 20:51:32.885772 1163 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:32.887794 1745 consensus_queue.cc:1035] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:32.882730 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:32.884200 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4e304a1212412f8e99bfd222a110b5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.885353 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:32.889010 1044 raft_consensus.cc:3054] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:32.889330 983 leader_election.cc:343] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 0e2235e2a63c4e1192b3260085908445
I20250112 20:51:32.889518 983 leader_election.cc:304] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:32.889992 1821 raft_consensus.cc:3054] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.891961 1044 raft_consensus.cc:2463] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.893947 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.894326 1178 tablet_bootstrap.cc:654] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.895165 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:32.895455 1185 log.cc:1198] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/31d3669355bc4b6bbc665edaa80c7a9e
I20250112 20:51:32.895870 1185 ts_tablet_manager.cc:1939] T 31d3669355bc4b6bbc665edaa80c7a9e P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:32.896277 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 301) took 1039 ms. Trace:
I20250112 20:51:32.909335 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.909608 1178 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:32.909754 1178 ts_tablet_manager.cc:1397] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20250112 20:51:32.910532 1178 raft_consensus.cc:357] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:32.910773 1178 raft_consensus.cc:383] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:32.896694 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a9e6cf325d48cc9a2a956400a68deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:32.910889 1178 raft_consensus.cc:738] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:32.911022 772 raft_consensus.cc:3054] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:32.911172 1178 consensus_queue.cc:260] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:32.911633 1178 ts_tablet_manager.cc:1428] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:32.912401 1178 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:32.896898 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4e304a1212412f8e99bfd222a110b5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.896929 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c8dc85ac9240598d6e413deac19abe" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:32.913264 769 raft_consensus.cc:2388] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:32.897131 744 tablet_service.cc:1467] Processing CreateTablet for tablet c8d24bccf682483aa9b273fc1ce847da (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.914286 772 raft_consensus.cc:2463] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:32.914297 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d24bccf682483aa9b273fc1ce847da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.897229 1164 consensus_queue.cc:260] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.917973 1178 tablet_bootstrap.cc:654] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.918148 1638 raft_consensus.cc:491] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:32.918315 1638 raft_consensus.cc:513] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:32.918800 1164 ts_tablet_manager.cc:1428] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.305s user 0.001s sys 0.002s
W20250112 20:51:32.900002 755 consensus_meta.cc:287] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: Time spent flushing consensus metadata: real 0.517s user 0.004s sys 0.001s
I20250112 20:51:32.904737 739 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d3722fd14d59b9994655cd30ef2b (DEFAULT_TABLE table=default.Table_a6951a044a6446d9b2a2bdd8d43f305b [id=fdfa5636686548deb94302c32d7a1757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.906984 1780 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.922240 1185 rpcz_store.cc:276] 0112 20:51:31.856416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.856977 (+ 561us) service_pool.cc:225] Handling call
0112 20:51:32.527601 (+670624us) ts_tablet_manager.cc:1145] Deleting tablet 31d3669355bc4b6bbc665edaa80c7a9e
0112 20:51:32.527613 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:32.723293 (+195680us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:32.778306 (+ 55013us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:32.896235 (+117929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":670385}
I20250112 20:51:32.922961 1185 tablet_replica.cc:331] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:32.924566 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f11304909b416aa1e9db1abf633a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.887987 1761 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:32.928251 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:32.892874 1821 raft_consensus.cc:2743] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:32.820569 1184 ts_tablet_manager.cc:1918] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:32.928956 1184 log.cc:1198] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b32dc02497f64a2e9ad99da6c9115d5d
I20250112 20:51:32.929458 1184 ts_tablet_manager.cc:1939] T b32dc02497f64a2e9ad99da6c9115d5d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:32.929423 1846 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:32.930343 1184 tablet_replica.cc:331] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:32.930567 1184 raft_consensus.cc:2238] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.930819 1184 pending_rounds.cc:62] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:32.930900 1184 pending_rounds.cc:69] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585015732027392 op_type: NO_OP noop_request { }
I20250112 20:51:32.931027 1184 raft_consensus.cc:2883] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:32.931139 1184 raft_consensus.cc:2267] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.931671 1184 ts_tablet_manager.cc:1905] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.931699 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.932328 881 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d3722fd14d59b9994655cd30ef2b (DEFAULT_TABLE table=default.Table_a6951a044a6446d9b2a2bdd8d43f305b [id=fdfa5636686548deb94302c32d7a1757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.932730 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc90d3722fd14d59b9994655cd30ef2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.896477 1163 tablet_bootstrap.cc:654] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:32.901716 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e72a6c8e18934b789a048d5aaded121b (DEFAULT_TABLE table=default.Table_284c5bfc11f1458dbade76ac78b87f20 [id=f83f83db15594cd9928fccf547cad7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.901718 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (DEFAULT_TABLE table=default.Table_b2428906915d47bd9fb06b88e971194c [id=fb7c117897c24636b7e2dd8109e782d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.901969 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 56544c3b0cf140089baa678765a2eb79 (DEFAULT_TABLE table=default.Table_58f5707fbcdd4c339243ab9c0645d2aa [id=462ba020c7414465b636c07442083e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.902118 1022 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f3389061143509354f37ae93ef0c7 (DEFAULT_TABLE table=default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.902369 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 53551931b23d43da89e9193cdf034a63 (DEFAULT_TABLE table=default.Table_c5469ad8d235490ebfbbb933c659d6ec [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.934134 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56544c3b0cf140089baa678765a2eb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.934670 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53551931b23d43da89e9193cdf034a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.904533 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b930069b6f1b49439dbc0a7abd3fba72 (DEFAULT_TABLE table=default.Table_29c0f892c9954a598f7f27c4be906a91 [id=61af9ec286544b919c9f053b647bc6fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.935231 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4ef01b41b6140d6bdad07bc79dd7dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.940980 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a6c8e18934b789a048d5aaded121b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.941288 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:32.941710 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b930069b6f1b49439dbc0a7abd3fba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.923838 1663 consensus_queue.cc:1035] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:32.942339 847 consensus_peers.cc:487] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:32.944752 847 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.945649 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2f3389061143509354f37ae93ef0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.945689 1185 raft_consensus.cc:2238] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:32.946166 1185 raft_consensus.cc:2267] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:32.946789 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.947178 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d97ac93467a4a7a8ae44d52719f191c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.947337 1185 ts_tablet_manager.cc:1905] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:32.946599 847 leader_election.cc:304] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:32.948556 1817 raft_consensus.cc:2798] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:32.948720 1817 raft_consensus.cc:695] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:32.949019 1817 consensus_queue.cc:237] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:32.954110 847 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:32.956583 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 (DEFAULT_TABLE table=default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.956827 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 (DEFAULT_TABLE table=default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.956992 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd47d1c1a4b42bdbff97f619dc95134. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.957219 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd34e829d0645ea80daaff5d27cb266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.957456 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.957681 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4421d16e3f423794c44970a2469b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.957906 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2235e2a63c4e1192b3260085908445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.958127 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.958320 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f11304909b416aa1e9db1abf633a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.958549 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.958889 1014 tablet_service.cc:1467] Processing CreateTablet for tablet cc90d3722fd14d59b9994655cd30ef2b (DEFAULT_TABLE table=default.Table_a6951a044a6446d9b2a2bdd8d43f305b [id=fdfa5636686548deb94302c32d7a1757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.959250 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc90d3722fd14d59b9994655cd30ef2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.959358 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.959561 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.959877 1010 tablet_service.cc:1467] Processing CreateTablet for tablet c8d24bccf682483aa9b273fc1ce847da (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.960019 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 (DEFAULT_TABLE table=default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.960211 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d24bccf682483aa9b273fc1ce847da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.960542 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8229e4ef771b4032a4bfae3190a851e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.960574 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ade9b410d8443249730cacdf7266eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.960803 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.961042 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d3669355bc4b6bbc665edaa80c7a9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6092a31de9bc47e4bae5f5aebb180c6d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.961282 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.961477 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7a54ce78854137ab2d12e9ce055565 with delete_type TABLET_DATA_DELETED (replaced by tablet 7293510f40804e2aabccf92986489dab) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.961678 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.960814 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 158cfe5ba52c4771af97841885b29f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:32.956768 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 (DEFAULT_TABLE table=default.Table_6c73da0cad5b49beb8a7ec45996a6d6d [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.964138 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b8cc99742f4f9ba68eb1aa17cab248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.959231 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab (DEFAULT_TABLE table=default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:32.965636 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5b10576bd34af5b02f8d8dcc3759ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.919211 1638 leader_election.cc:290] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:32.924417 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc90d3722fd14d59b9994655cd30ef2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:32.924858 1164 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:32.940040 710 leader_election.cc:343] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:32.989804 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 49691.4us
I20250112 20:51:32.885048 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 158cfe5ba52c4771af97841885b29f9d (table default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]) successfully deleted
W20250112 20:51:32.996338 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 111304us
I20250112 20:51:32.965977 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921] alter complete (version 3)
I20250112 20:51:32.997942 1164 tablet_bootstrap.cc:654] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.000097 1184 ts_tablet_manager.cc:1918] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.001235 1184 log.cc:1198] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/bcbbefa7b55041a6a6b50b3c96c45833
I20250112 20:51:33.001929 1184 ts_tablet_manager.cc:1939] T bcbbefa7b55041a6a6b50b3c96c45833 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.002573 1185 ts_tablet_manager.cc:1918] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.004184 1184 tablet_replica.cc:331] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.005463 1185 log.cc:1198] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3ade9b410d8443249730cacdf7266eb2
I20250112 20:51:33.006330 1185 ts_tablet_manager.cc:1939] T 3ade9b410d8443249730cacdf7266eb2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.011397 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a6951a044a6446d9b2a2bdd8d43f305b" } new_table_name: "default.Table_11d93ce9bed540a6b14e29a387b232ba" modify_external_catalogs: true
I20250112 20:51:33.012919 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 302) took 1155 ms. Trace:
I20250112 20:51:33.013370 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8eeab516a6794571a8336068218e7a3a" } modify_external_catalogs: true
I20250112 20:51:33.013484 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ca061b82a894e91b9508613ccd8ecd7 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.013594 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8eeab516a6794571a8336068218e7a3a" } modify_external_catalogs: true
I20250112 20:51:33.013892 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15828ef1bbbc41068bb6b6f2fbb18f94" } modify_external_catalogs: true
I20250112 20:51:33.014098 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15828ef1bbbc41068bb6b6f2fbb18f94" } modify_external_catalogs: true
I20250112 20:51:33.014256 1184 raft_consensus.cc:2238] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.014425 1184 raft_consensus.cc:2267] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.014997 1184 ts_tablet_manager.cc:1905] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.014999 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0d195bfd940c4601bf584ba32f274f76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.015486 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d195bfd940c4601bf584ba32f274f76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.015447 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a77dee800e62415db3cada3902c1efcc" } modify_external_catalogs: true
I20250112 20:51:33.015632 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a77dee800e62415db3cada3902c1efcc" } modify_external_catalogs: true
I20250112 20:51:33.013370 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_09bd1e992b194beb825bb3280677c608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.016139 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09bd1e992b194beb825bb3280677c608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.016219 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_714de3355adc4778a417b60c786e8e4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.016640 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_714de3355adc4778a417b60c786e8e4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.017650 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c456c3048d45444f9422bf7a190b290a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.018138 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c456c3048d45444f9422bf7a190b290a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.019398 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_326b5443cb3245d0bafc2e553c0bfda8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:33.019727 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:33.019827 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_326b5443cb3245d0bafc2e553c0bfda8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.020313 1046 raft_consensus.cc:1270] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Refusing update from remote peer aefe1db8b19244d891b6bcd8971e9b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:33.023322 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.024885 1163 tablet_bootstrap.cc:492] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.025038 1163 ts_tablet_manager.cc:1397] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.004s
I20250112 20:51:33.025789 1163 raft_consensus.cc:357] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.025985 1163 raft_consensus.cc:383] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.026054 1163 raft_consensus.cc:738] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.026304 1163 consensus_queue.cc:260] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:33.026770 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc90d3722fd14d59b9994655cd30ef2b consensus state has no leader
I20250112 20:51:33.026861 1163 ts_tablet_manager.cc:1428] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:33.027132 1163 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:33.021158 710 leader_election.cc:343] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.027987 710 leader_election.cc:304] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:33.028316 1638 raft_consensus.cc:2743] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.028615 1638 consensus_queue.cc:1035] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.027959 1185 rpcz_store.cc:276] 0112 20:51:31.857370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.857580 (+ 210us) service_pool.cc:225] Handling call
0112 20:51:32.922876 (+1065296us) ts_tablet_manager.cc:1145] Deleting tablet 3ade9b410d8443249730cacdf7266eb2
0112 20:51:32.922885 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:32.945993 (+ 23108us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x616000162154. stack: 00007f1a9e448fd6 00007f1a9e448d90 00007f1a9d42848d 000000000096d544 000000000096d50d 000000000096ceda 00007f1aa9cebed0 00007f1aaafe0463 00007f1aada84eb7 00007f1aadab7584 00007f1aada98690 00007f1aada9841d 0000000000a71ecc 00007f1a9e48fb64 00007f1a9e49318b 00007f1a9e493008
0112 20:51:32.988452 (+ 42459us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.002036 (+ 13584us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.012875 (+ 10839us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6823296,"tablet-delete.queue_time_us":1065118}
I20250112 20:51:33.029625 1185 tablet_replica.cc:331] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:33.029902 1185 raft_consensus.cc:2238] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.030088 1185 pending_rounds.cc:62] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:33.030169 1185 pending_rounds.cc:69] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585015732027392 op_type: NO_OP noop_request { }
I20250112 20:51:33.030296 1185 raft_consensus.cc:2883] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:33.030429 1185 raft_consensus.cc:2267] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.031698 1185 ts_tablet_manager.cc:1905] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.032150 847 leader_election.cc:304] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.032994 1841 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:33.033872 1790 raft_consensus.cc:491] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.034013 1832 raft_consensus.cc:2743] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:33.034129 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 78650.5us
I20250112 20:51:33.034603 847 leader_election.cc:304] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.039743 1846 tablet.cc:1721] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.041296 1045 raft_consensus.cc:3054] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.042009 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b32dc02497f64a2e9ad99da6c9115d5d (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:33.043401 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet bcbbefa7b55041a6a6b50b3c96c45833 (table default.Table_69153618434846ad9d90837893bea2da [id=be8442e96b3b4635ba6dac3956ee9de3]) successfully deleted
I20250112 20:51:33.043793 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3ade9b410d8443249730cacdf7266eb2 (table default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]) successfully deleted
I20250112 20:51:33.045001 1780 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:33.045042 1761 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:33.046908 739 tablet_service.cc:1467] Processing CreateTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd (DEFAULT_TABLE table=default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.047276 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924082716c1e4d9ea6fbe97ae93232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.049172 754 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d (DEFAULT_TABLE table=default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.049513 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f807071e53a4f0e8629b1dfa6d26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.051044 752 tablet_service.cc:1467] Processing CreateTablet for tablet dea98c29bc744009ab9b3eeb53684f29 (DEFAULT_TABLE table=default.Table_0cbb400711764b7a8fffe933e3334d42 [id=cfaa5f6221c1491b822a803a83e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.052739 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5ca061b82a894e91b9508613ccd8ecd7
I20250112 20:51:33.053256 450 catalog_manager.cc:6596] ReplaceTablet: tablet 5ca061b82a894e91b9508613ccd8ecd7 deleted and replaced by tablet 1b1565c7487c428f8f963aca406263fe
I20250112 20:51:33.053293 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3bbb679585fc44699553fe85ea044566
I20250112 20:51:33.054204 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b2ca205a6c6f470c896d5194717dbf72" } new_table_name: "default.Table_28283ef30e624ee992db4e187bd92df0" modify_external_catalogs: true
I20250112 20:51:33.055321 450 catalog_manager.cc:5526] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:33.055783 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 24c8dc85ac9240598d6e413deac19abe (table default.Table_b2ca205a6c6f470c896d5194717dbf72 [id=3a4de1859ffd4db3931ce68febdd8836]). Expected version 2 got 0
I20250112 20:51:33.055790 1825 tablet.cc:1721] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.056180 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea98c29bc744009ab9b3eeb53684f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.056232 1790 raft_consensus.cc:513] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.056895 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:33.057580 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_43ae987b5bce459c8214a0ea34b9df01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.058077 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ae987b5bce459c8214a0ea34b9df01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:33.059871 710 leader_election.cc:343] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.060010 710 leader_election.cc:304] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:33.060266 1638 raft_consensus.cc:2743] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.060634 1638 consensus_queue.cc:1035] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.061147 755 tablet_service.cc:1467] Processing CreateTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.061471 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669e1cfd407e4d87a63e6832d9c6fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.061779 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 3dba3d9315db4b9292288ee62c2244bc (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.062294 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dba3d9315db4b9292288ee62c2244bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.064893 1029 tablet_service.cc:1467] Processing CreateTablet for tablet dc3a9740415140fe8864d54f477af954 (DEFAULT_TABLE table=default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.065304 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc3a9740415140fe8864d54f477af954. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.068727 1831 raft_consensus.cc:2743] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.069288 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.069316 746 tablet_service.cc:1467] Processing CreateTablet for tablet dc3a9740415140fe8864d54f477af954 (DEFAULT_TABLE table=default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.069638 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669e1cfd407e4d87a63e6832d9c6fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.069636 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc3a9740415140fe8864d54f477af954. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.069996 744 tablet_service.cc:1467] Processing CreateTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 (DEFAULT_TABLE table=default.Table_b3533fddf0f8403bb052fb6b07ab0b20 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.070307 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2664a2e98f74f73ab2dc0adaa2335d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.070539 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd (DEFAULT_TABLE table=default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.071477 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 557fa6445eee481dbc4076f67fe63466 (DEFAULT_TABLE table=default.Table_aa5b2f32e08b4d1d8c28904dbda61f18 [id=727fcc0dcec04c1f9b7fbc5956c2e796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.071918 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924082716c1e4d9ea6fbe97ae93232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.076172 749 tablet_service.cc:1467] Processing CreateTablet for tablet 557fa6445eee481dbc4076f67fe63466 (DEFAULT_TABLE table=default.Table_aa5b2f32e08b4d1d8c28904dbda61f18 [id=727fcc0dcec04c1f9b7fbc5956c2e796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.076524 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fa6445eee481dbc4076f67fe63466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.076457 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d (DEFAULT_TABLE table=default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.076822 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f807071e53a4f0e8629b1dfa6d26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.077230 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fa6445eee481dbc4076f67fe63466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.078835 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f9f2cdcb247848539893c00bfbef8b18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.079372 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9f2cdcb247848539893c00bfbef8b18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.083467 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3dba3d9315db4b9292288ee62c2244bc (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.083838 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dba3d9315db4b9292288ee62c2244bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.083828 881 tablet_service.cc:1467] Processing CreateTablet for tablet c8d24bccf682483aa9b273fc1ce847da (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.089205 879 tablet_service.cc:1467] Processing CreateTablet for tablet 98547168378947df885c601ccdc85e57 (DEFAULT_TABLE table=default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.089391 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_23317c504a8b4a7392676291f5385b74" } new_table_name: "default.Table_f53bd3309a434176b88eb5b8b727b344" modify_external_catalogs: true
I20250112 20:51:33.089396 1014 tablet_service.cc:1467] Processing CreateTablet for tablet dea98c29bc744009ab9b3eeb53684f29 (DEFAULT_TABLE table=default.Table_0cbb400711764b7a8fffe933e3334d42 [id=cfaa5f6221c1491b822a803a83e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.089787 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea98c29bc744009ab9b3eeb53684f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.092101 878 tablet_service.cc:1467] Processing CreateTablet for tablet 557fa6445eee481dbc4076f67fe63466 (DEFAULT_TABLE table=default.Table_aa5b2f32e08b4d1d8c28904dbda61f18 [id=727fcc0dcec04c1f9b7fbc5956c2e796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.092645 1186 raft_consensus.cc:2238] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.092684 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ebead212ebdd45e29399da06bdace3c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.093312 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebead212ebdd45e29399da06bdace3c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.093255 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3dba3d9315db4b9292288ee62c2244bc (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.093909 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d24bccf682483aa9b273fc1ce847da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.094347 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98547168378947df885c601ccdc85e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.097553 886 tablet_service.cc:1467] Processing CreateTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd (DEFAULT_TABLE table=default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.098125 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dba3d9315db4b9292288ee62c2244bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.100919 1186 raft_consensus.cc:2267] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.094187 1164 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.101562 1164 ts_tablet_manager.cc:1397] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.182s user 0.004s sys 0.001s
I20250112 20:51:33.102458 1164 raft_consensus.cc:357] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.103302 1186 ts_tablet_manager.cc:1905] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.104533 1790 leader_election.cc:290] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.105795 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fa6445eee481dbc4076f67fe63466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.107386 876 tablet_service.cc:1467] Processing CreateTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 (DEFAULT_TABLE table=default.Table_b3533fddf0f8403bb052fb6b07ab0b20 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.108001 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924082716c1e4d9ea6fbe97ae93232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.105146 1638 raft_consensus.cc:491] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.109184 1638 raft_consensus.cc:513] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.109871 1638 leader_election.cc:290] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.094597 751 tablet_service.cc:1467] Processing CreateTablet for tablet 98547168378947df885c601ccdc85e57 (DEFAULT_TABLE table=default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.110170 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2664a2e98f74f73ab2dc0adaa2335d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.110455 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.110903 1164 raft_consensus.cc:383] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.111373 1164 raft_consensus.cc:738] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.113297 912 raft_consensus.cc:2463] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:33.113724 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d (DEFAULT_TABLE table=default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.114424 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f807071e53a4f0e8629b1dfa6d26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.114027 710 leader_election.cc:304] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.115152 1638 raft_consensus.cc:2798] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.115259 1638 raft_consensus.cc:491] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.115360 1638 raft_consensus.cc:3054] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.116636 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98547168378947df885c601ccdc85e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.116583 877 tablet_service.cc:1467] Processing CreateTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f (DEFAULT_TABLE table=default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.117882 1638 raft_consensus.cc:513] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.118876 1164 consensus_queue.cc:260] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.119032 887 tablet_service.cc:1467] Processing CreateTablet for tablet dc3a9740415140fe8864d54f477af954 (DEFAULT_TABLE table=default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.120419 880 tablet_service.cc:1467] Processing CreateTablet for tablet dea98c29bc744009ab9b3eeb53684f29 (DEFAULT_TABLE table=default.Table_0cbb400711764b7a8fffe933e3334d42 [id=cfaa5f6221c1491b822a803a83e8f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.121744 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc3a9740415140fe8864d54f477af954. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.122406 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea98c29bc744009ab9b3eeb53684f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.124109 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.127358 912 raft_consensus.cc:3054] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.131219 912 raft_consensus.cc:2463] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:33.131601 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669e1cfd407e4d87a63e6832d9c6fa1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.120461 1780 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:33.132879 710 leader_election.cc:304] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.122460 1638 leader_election.cc:290] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.136739 1780 tablet.cc:1721] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.137838 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.141794 776 raft_consensus.cc:2463] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:33.137992 1745 raft_consensus.cc:2798] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.142226 1745 raft_consensus.cc:695] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:33.142652 1745 consensus_queue.cc:237] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.145151 1663 raft_consensus.cc:491] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.145315 1663 raft_consensus.cc:513] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.146008 1663 leader_election.cc:290] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:33.138387 1164 ts_tablet_manager.cc:1428] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250112 20:51:33.150017 1164 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.152516 1164 tablet_bootstrap.cc:654] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.152992 1831 consensus_queue.cc:1035] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250112 20:51:33.156134 1860 tablet.cc:1721] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.156530 1831 consensus_queue.cc:1035] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.157302 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.162122 1841 tablet.cc:1721] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.163309 1018 tablet_service.cc:1467] Processing CreateTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 (DEFAULT_TABLE table=default.Table_b3533fddf0f8403bb052fb6b07ab0b20 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.163831 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2664a2e98f74f73ab2dc0adaa2335d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.166550 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b1565c7487c428f8f963aca406263fe) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.166810 1163 tablet_bootstrap.cc:654] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.167285 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 98547168378947df885c601ccdc85e57 (DEFAULT_TABLE table=default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.167650 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98547168378947df885c601ccdc85e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.170540 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.171749 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.172122 1825 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.174773 847 leader_election.cc:304] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.189132 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.197685 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.198190 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:33.200029 1866 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.202373 1045 raft_consensus.cc:2463] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:33.208927 1780 tablet.cc:1721] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:33.212348 710 leader_election.cc:343] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:33.212682 710 leader_election.cc:343] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:33.212950 710 leader_election.cc:343] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.218451 1185 ts_tablet_manager.cc:1918] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.227174 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5f012db68caf4d7d86afa6deb9f2ffaa
I20250112 20:51:33.228376 1831 raft_consensus.cc:2798] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.228287 1704 raft_consensus.cc:491] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.231622 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b1565c7487c428f8f963aca406263fe) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.232645 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.234030 1178 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.234290 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.237082 1704 raft_consensus.cc:513] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.237986 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.238679 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.239758 1831 raft_consensus.cc:491] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.241232 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.246969 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_361fe8fc7b6845369a0ee99bc93d2771" } modify_external_catalogs: true
I20250112 20:51:33.247191 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_361fe8fc7b6845369a0ee99bc93d2771" } modify_external_catalogs: true
I20250112 20:51:33.249465 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.232385 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:33.243732 710 leader_election.cc:343] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.254072 710 leader_election.cc:304] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:33.254635 1745 raft_consensus.cc:2743] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.254962 1185 log.cc:1198] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/bcbbefa7b55041a6a6b50b3c96c45833
I20250112 20:51:33.256465 1185 ts_tablet_manager.cc:1939] T bcbbefa7b55041a6a6b50b3c96c45833 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.258076 1831 raft_consensus.cc:3054] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.258385 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.259586 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca061b82a894e91b9508613ccd8ecd7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b1565c7487c428f8f963aca406263fe) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.267037 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 303) took 1408 ms. Trace:
I20250112 20:51:33.268041 1178 ts_tablet_manager.cc:1397] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.322s user 0.020s sys 0.010s
W20250112 20:51:33.273100 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24c8dc85ac9240598d6e413deac19abe consensus state has no leader
I20250112 20:51:33.274093 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f012db68caf4d7d86afa6deb9f2ffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.274925 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f79972c8942460f988e2d87f374fc5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.275481 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f79972c8942460f988e2d87f374fc5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.277912 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e6cec6d0054141a6a47a840892189740" } new_table_name: "default.Table_8a830df519164b7b89e43fa9335a1a20" modify_external_catalogs: true
I20250112 20:51:33.278594 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.282223 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]). Expected version 1 got 0
W20250112 20:51:33.282305 847 consensus_peers.cc:487] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5af0bc83bc415b972cbbe0fb39f75f. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:33.283874 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:33.284108 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]). Expected version 2 got 0
I20250112 20:51:33.284457 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]). Expected version 1 got 0
I20250112 20:51:33.287066 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_58f5707fbcdd4c339243ab9c0645d2aa" } new_table_name: "default.Table_bc620e652af0401bb45d3cab54240f0c" modify_external_catalogs: true
I20250112 20:51:33.289076 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.289758 1761 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.257797 1164 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.290553 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.292763 1704 leader_election.cc:290] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.291074 1860 tablet.cc:1721] T c2a8f71233d04ca39e49932308e56492 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.294764 1164 ts_tablet_manager.cc:1397] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.144s user 0.001s sys 0.006s
I20250112 20:51:33.285161 1863 tablet.cc:1721] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.295836 1164 raft_consensus.cc:357] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.296177 1164 raft_consensus.cc:383] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.296308 1164 raft_consensus.cc:738] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.297797 1761 tablet.cc:1721] T c2a8f71233d04ca39e49932308e56492 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.283670 1178 raft_consensus.cc:357] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.300091 1164 consensus_queue.cc:260] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.301025 1817 consensus_queue.cc:1035] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 20:51:33.301836 1164 ts_tablet_manager.cc:1428] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250112 20:51:33.303004 1185 rpcz_store.cc:276] 0112 20:51:31.858091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.858277 (+ 186us) service_pool.cc:225] Handling call
0112 20:51:33.029541 (+1171264us) ts_tablet_manager.cc:1145] Deleting tablet bcbbefa7b55041a6a6b50b3c96c45833
0112 20:51:33.029555 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.165242 (+135687us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.218315 (+ 53073us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.266994 (+ 48679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171034}
I20250112 20:51:33.303603 1185 tablet_replica.cc:331] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:33.304000 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet bcbbefa7b55041a6a6b50b3c96c45833 (table default.Table_69153618434846ad9d90837893bea2da [id=be8442e96b3b4635ba6dac3956ee9de3]) successfully deleted
W20250112 20:51:33.305450 847 leader_election.cc:343] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.305428 1846 tablet.cc:1721] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.308683 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5401cf6f9421453fbe9e59937a098182" } modify_external_catalogs: true
I20250112 20:51:33.308900 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5401cf6f9421453fbe9e59937a098182" } modify_external_catalogs: true
I20250112 20:51:33.309024 1186 ts_tablet_manager.cc:1918] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.312942 1185 raft_consensus.cc:2238] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.313221 1185 raft_consensus.cc:2267] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.312920 1178 raft_consensus.cc:383] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.314431 1178 raft_consensus.cc:738] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:33.317134 847 leader_election.cc:343] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.317194 1184 ts_tablet_manager.cc:1918] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.317890 1761 tablet.cc:1721] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.322219 1186 log.cc:1198] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/0e2235e2a63c4e1192b3260085908445
I20250112 20:51:33.322597 1186 ts_tablet_manager.cc:1939] T 0e2235e2a63c4e1192b3260085908445 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:33.323168 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_31a6df3a24c042a38ee31d789e5f46eb" } new_table_name: "default.Table_42c75ef9c5564cfb835f9aa1a3ad4ecb" modify_external_catalogs: true
I20250112 20:51:33.324347 1780 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.325359 1184 log.cc:1198] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8b414be57990432bb786daf5faf0cc2e
I20250112 20:51:33.327234 1178 consensus_queue.cc:260] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.319490 847 leader_election.cc:304] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.331017 1185 ts_tablet_manager.cc:1905] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.332041 1184 ts_tablet_manager.cc:1939] T 8b414be57990432bb786daf5faf0cc2e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.332444 1164 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.333204 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8b414be57990432bb786daf5faf0cc2e (table default.Table_80526018642b4dd5882083cb240f243d [id=7ca73b57a2da4859a58a0e5711463c39]) successfully deleted
I20250112 20:51:33.335419 1164 tablet_bootstrap.cc:654] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.337038 770 raft_consensus.cc:1270] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 30ba399c7a6749e0acd075d93e4cc4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:33.340598 1761 tablet.cc:1721] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.342841 1821 consensus_queue.cc:1035] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250112 20:51:33.343865 1163 tablet_bootstrap.cc:492] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.344131 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 317) took 1024 ms. Trace:
I20250112 20:51:33.344154 1163 ts_tablet_manager.cc:1397] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.317s user 0.004s sys 0.012s
I20250112 20:51:33.345516 1163 raft_consensus.cc:357] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.346233 1163 raft_consensus.cc:383] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.346398 1163 raft_consensus.cc:738] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.346815 1163 consensus_queue.cc:260] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.347759 1163 ts_tablet_manager.cc:1428] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:33.349124 1163 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.350920 1829 raft_consensus.cc:2743] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.352129 1704 raft_consensus.cc:491] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.352576 1178 ts_tablet_manager.cc:1428] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20250112 20:51:33.352813 1163 tablet_bootstrap.cc:654] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.353732 1184 tablet_replica.cc:331] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.355158 1841 tablet.cc:1721] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.355640 1184 raft_consensus.cc:2238] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.355942 1184 raft_consensus.cc:2267] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.356631 1184 ts_tablet_manager.cc:1905] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.356454 1186 rpcz_store.cc:276] 0112 20:51:32.298715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.299797 (+ 1082us) service_pool.cc:225] Handling call
0112 20:51:32.777955 (+478158us) ts_tablet_manager.cc:1145] Deleting tablet 0e2235e2a63c4e1192b3260085908445
0112 20:51:32.777965 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.219890 (+441925us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.299194 (+ 79304us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.322892 (+ 23698us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":290236,"tablet-delete.queue_time_us":478016}
I20250112 20:51:33.355823 1704 raft_consensus.cc:513] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.371130 912 raft_consensus.cc:1270] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Refusing update from remote peer 30ba399c7a6749e0acd075d93e4cc4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:33.371964 1821 consensus_queue.cc:1035] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.375481 1831 raft_consensus.cc:513] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.376168 1831 leader_election.cc:290] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.376672 1178 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:33.379887 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89f9b46da06a476ab9f2a93ca74ff736 consensus state has no leader
I20250112 20:51:33.380537 1178 tablet_bootstrap.cc:654] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.398268 1704 leader_election.cc:290] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:33.398515 1186 tablet_replica.cc:331] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:33.399156 1186 raft_consensus.cc:2238] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:33.399608 1186 pending_rounds.cc:62] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:33.399760 1186 pending_rounds.cc:69] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585017032568832 op_type: NO_OP noop_request { }
I20250112 20:51:33.399992 1186 raft_consensus.cc:2883] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:33.400175 1186 raft_consensus.cc:2267] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.400950 1186 ts_tablet_manager.cc:1905] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.405077 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:33.405288 775 raft_consensus.cc:3054] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.405331 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c86ad5f59d431e9b1a1fa890fd35c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:33.405409 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:33.405532 1047 raft_consensus.cc:3054] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.407310 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 0e2235e2a63c4e1192b3260085908445 (table default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]) successfully deleted
I20250112 20:51:33.408669 775 raft_consensus.cc:2463] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:33.409283 1704 raft_consensus.cc:491] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.409456 1704 raft_consensus.cc:513] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.409512 1047 raft_consensus.cc:2463] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:33.410149 1704 leader_election.cc:290] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.411242 1184 ts_tablet_manager.cc:1918] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.413540 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.413529 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.414440 1184 log.cc:1198] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7f39195945024f4fba9942faec70bc31
I20250112 20:51:33.414927 847 leader_election.cc:304] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.415253 1704 raft_consensus.cc:2798] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.415439 1704 raft_consensus.cc:695] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:33.415016 1184 ts_tablet_manager.cc:1939] T 7f39195945024f4fba9942faec70bc31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.415735 1704 consensus_queue.cc:237] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.416996 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a9245c5997d7416fa75ebf6202856645" } modify_external_catalogs: true
I20250112 20:51:33.417227 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a9245c5997d7416fa75ebf6202856645" } modify_external_catalogs: true
I20250112 20:51:33.418107 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
W20250112 20:51:33.419723 847 leader_election.cc:343] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:33.420473 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:33.420620 847 leader_election.cc:304] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.421456 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 323) took 1041 ms. Trace:
I20250112 20:51:33.421520 1831 raft_consensus.cc:2743] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.424634 1184 rpcz_store.cc:276] 0112 20:51:32.374864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.374902 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:33.333531 (+958629us) ts_tablet_manager.cc:1145] Deleting tablet 7f39195945024f4fba9942faec70bc31
0112 20:51:33.333544 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.392243 (+ 58699us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.411106 (+ 18863us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.416390 (+ 5284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":958422}
I20250112 20:51:33.425782 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7f39195945024f4fba9942faec70bc31 (table default.Table_a1d08a9783214c518f1eda3d907286a1 [id=f0b77d4d8d69417680f0fed74791df36]) successfully deleted
I20250112 20:51:33.426069 1184 tablet_replica.cc:331] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.426573 1184 raft_consensus.cc:2238] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.427125 1184 raft_consensus.cc:2267] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.429121 1825 tablet.cc:1721] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.427987 1184 ts_tablet_manager.cc:1905] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.438766 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7" } modify_external_catalogs: true
I20250112 20:51:33.439019 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7" } modify_external_catalogs: true
I20250112 20:51:33.445943 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:33.450752 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.451305 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:33.452334 847 leader_election.cc:343] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.453987 1761 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.458818 1780 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.459456 847 leader_election.cc:304] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.461092 1164 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.463143 1587 raft_consensus.cc:2743] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.468739 1164 ts_tablet_manager.cc:1397] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.002s
I20250112 20:51:33.472563 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eadef6eb5b6841f88001ccdad8f9ab3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.473230 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eadef6eb5b6841f88001ccdad8f9ab3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.473801 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.482532 1184 ts_tablet_manager.cc:1918] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.482702 1184 log.cc:1198] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ef89229192934339954566e366b6891c
I20250112 20:51:33.483114 1184 ts_tablet_manager.cc:1939] T ef89229192934339954566e366b6891c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.483079 1857 consensus_queue.cc:1035] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250112 20:51:33.484205 1164 raft_consensus.cc:357] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.484424 1164 raft_consensus.cc:383] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.484473 1163 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.484515 1164 raft_consensus.cc:738] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.484622 1163 ts_tablet_manager.cc:1397] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.136s user 0.012s sys 0.008s
I20250112 20:51:33.484737 1164 consensus_queue.cc:260] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.485352 1164 ts_tablet_manager.cc:1428] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250112 20:51:33.485622 1164 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.485456 1163 raft_consensus.cc:357] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.486033 1163 raft_consensus.cc:383] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.486110 1163 raft_consensus.cc:738] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.486667 1163 consensus_queue.cc:260] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.488234 1163 ts_tablet_manager.cc:1428] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:33.494777 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 324) took 1119 ms. Trace:
I20250112 20:51:33.494885 1184 rpcz_store.cc:276] 0112 20:51:32.375254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.375297 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:33.425983 (+1050686us) ts_tablet_manager.cc:1145] Deleting tablet ef89229192934339954566e366b6891c
0112 20:51:33.425999 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.445209 (+ 19210us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.482401 (+ 37192us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.494714 (+ 12313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3968,"tablet-delete.queue_time_us":1050384}
I20250112 20:51:33.495276 1184 tablet_replica.cc:331] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.495383 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ef89229192934339954566e366b6891c (table default.Table_f4fa0e0356d441bd8aa0cf8ae440f46e [id=e0268d57dc9647009d23c78817de46ff]) successfully deleted
I20250112 20:51:33.495465 1184 raft_consensus.cc:2238] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.495635 1184 raft_consensus.cc:2267] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.496821 1164 tablet_bootstrap.cc:654] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.499217 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_61b014e293d64e85a96563251b9229f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.499985 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b014e293d64e85a96563251b9229f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.500373 1184 ts_tablet_manager.cc:1905] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.502223 1163 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.504149 1163 tablet_bootstrap.cc:654] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:33.506771 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dc920e92de545fc961f82e7d0de28cd consensus state has no leader
I20250112 20:51:33.518134 1178 tablet_bootstrap.cc:492] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.522197 1178 ts_tablet_manager.cc:1397] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.165s user 0.005s sys 0.002s
I20250112 20:51:33.524667 1184 ts_tablet_manager.cc:1918] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.524912 1184 log.cc:1198] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/29b70ffb3e3341d398d36ff76993e9ac
I20250112 20:51:33.526382 1871 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:33.526994 1184 ts_tablet_manager.cc:1939] T 29b70ffb3e3341d398d36ff76993e9ac P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.528079 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 325) took 1152 ms. Trace:
I20250112 20:51:33.528272 1184 rpcz_store.cc:276] 0112 20:51:32.375648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.375686 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:33.495185 (+1119499us) ts_tablet_manager.cc:1145] Deleting tablet 29b70ffb3e3341d398d36ff76993e9ac
0112 20:51:33.495194 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.511632 (+ 16438us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.524524 (+ 12892us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.528025 (+ 3501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119306}
I20250112 20:51:33.529244 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 326) took 1153 ms. Trace:
I20250112 20:51:33.530318 1184 rpcz_store.cc:276] 0112 20:51:32.376035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.376076 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:33.529112 (+1153036us) ts_tablet_manager.cc:1145] Deleting tablet 2afa5b0717df4f83a2b44efac12fc06b
0112 20:51:33.529123 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.529211 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152889}
I20250112 20:51:33.531319 1184 tablet_replica.cc:331] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:33.531502 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2afa5b0717df4f83a2b44efac12fc06b because the tablet was not found. No further retry: Not found: Tablet not found: 2afa5b0717df4f83a2b44efac12fc06b
I20250112 20:51:33.531894 1184 raft_consensus.cc:2238] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.532519 1184 raft_consensus.cc:2267] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.533282 1184 ts_tablet_manager.cc:1905] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.534718 1178 raft_consensus.cc:357] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.534989 1178 raft_consensus.cc:383] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.535108 1178 raft_consensus.cc:738] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.535387 1178 consensus_queue.cc:260] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.535966 1178 ts_tablet_manager.cc:1428] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.000s sys 0.000s
I20250112 20:51:33.536382 1178 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:33.538988 1178 tablet_bootstrap.cc:654] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.544224 1163 tablet_bootstrap.cc:492] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.545138 1163 ts_tablet_manager.cc:1397] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.002s
I20250112 20:51:33.546164 1163 raft_consensus.cc:357] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.551108 1163 raft_consensus.cc:383] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.551498 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f020d2e2fdf24b08ba8be0a7cfb00340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:33.551826 1163 raft_consensus.cc:738] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:33.552201 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.562911 1701 raft_consensus.cc:491] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.563088 1701 raft_consensus.cc:513] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.563817 1186 ts_tablet_manager.cc:1918] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.564200 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72fda1743fc4f96b29add6e36e884de" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.564438 774 raft_consensus.cc:2463] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:33.565507 1701 leader_election.cc:290] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:33.565483 847 leader_election.cc:304] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.566159 1701 raft_consensus.cc:2798] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.566777 1701 raft_consensus.cc:491] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.566888 1701 raft_consensus.cc:3054] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.569803 1186 log.cc:1198] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/31d3669355bc4b6bbc665edaa80c7a9e
I20250112 20:51:33.570834 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72fda1743fc4f96b29add6e36e884de" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.571421 1163 consensus_queue.cc:260] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:33.571641 847 leader_election.cc:343] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.572057 1186 ts_tablet_manager.cc:1939] T 31d3669355bc4b6bbc665edaa80c7a9e P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:33.574340 1178 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.574811 1178 ts_tablet_manager.cc:1397] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250112 20:51:33.575502 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 318) took 1276 ms. Trace:
I20250112 20:51:33.576023 1186 rpcz_store.cc:276] 0112 20:51:32.298806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.300497 (+ 1691us) service_pool.cc:225] Handling call
0112 20:51:33.356842 (+1056345us) ts_tablet_manager.cc:1145] Deleting tablet 31d3669355bc4b6bbc665edaa80c7a9e
0112 20:51:33.356853 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.515320 (+158467us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.556733 (+ 41413us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.575131 (+ 18398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056177}
I20250112 20:51:33.576629 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 31d3669355bc4b6bbc665edaa80c7a9e (table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]) successfully deleted
I20250112 20:51:33.577158 1186 tablet_replica.cc:331] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:33.577368 1186 raft_consensus.cc:2238] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.578897 1186 raft_consensus.cc:2267] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.579456 1186 ts_tablet_manager.cc:1905] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.580466 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_77fce9ea810348bea460274d6f69db14 [id=9a97eeefbfac4dc2bb4a8ceab84304b3] alter complete (version 1)
I20250112 20:51:33.580664 1185 ts_tablet_manager.cc:1918] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.580816 1185 log.cc:1198] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/158cfe5ba52c4771af97841885b29f9d
I20250112 20:51:33.581290 1185 ts_tablet_manager.cc:1939] T 158cfe5ba52c4771af97841885b29f9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.587081 1184 ts_tablet_manager.cc:1918] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.587649 1184 log.cc:1198] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/cc971026f6b44db6af8215890ba71075
I20250112 20:51:33.587845 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 304) took 1728 ms. Trace:
I20250112 20:51:33.588001 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbbefa7b55041a6a6b50b3c96c45833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.588090 1163 ts_tablet_manager.cc:1428] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20250112 20:51:33.588363 1184 ts_tablet_manager.cc:1939] T cc971026f6b44db6af8215890ba71075 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.589756 1163 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.576038 1701 raft_consensus.cc:513] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.588006 1185 rpcz_store.cc:276] 0112 20:51:31.858822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.859046 (+ 224us) service_pool.cc:225] Handling call
0112 20:51:33.303542 (+1444496us) ts_tablet_manager.cc:1145] Deleting tablet 158cfe5ba52c4771af97841885b29f9d
0112 20:51:33.303551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.431694 (+128143us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.580547 (+148853us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.587512 (+ 6965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444321}
I20250112 20:51:33.592495 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 158cfe5ba52c4771af97841885b29f9d (table default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]) successfully deleted
I20250112 20:51:33.592849 1821 raft_consensus.cc:491] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.592998 1821 raft_consensus.cc:513] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.593571 1821 leader_election.cc:290] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.594352 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 327) took 1217 ms. Trace:
I20250112 20:51:33.594409 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.594516 1184 rpcz_store.cc:276] 0112 20:51:32.376390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.376431 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:33.531253 (+1154822us) ts_tablet_manager.cc:1145] Deleting tablet cc971026f6b44db6af8215890ba71075
0112 20:51:33.531262 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.543497 (+ 12235us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.586940 (+ 43443us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.594300 (+ 7360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154609}
I20250112 20:51:33.594899 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet cc971026f6b44db6af8215890ba71075 (table default.Table_0fa5226c1b51464da6a1b54f22ef816e [id=fbe29ec71ef847449805aeaa9f4a1220]) successfully deleted
I20250112 20:51:33.594913 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72fda1743fc4f96b29add6e36e884de" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:33.595106 774 raft_consensus.cc:3054] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.595629 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 329) took 1214 ms. Trace:
I20250112 20:51:33.595726 1184 rpcz_store.cc:276] 0112 20:51:32.380980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.381030 (+ 50us) service_pool.cc:225] Handling call
0112 20:51:33.594929 (+1213899us) ts_tablet_manager.cc:1145] Deleting tablet 83c13126e53b46c0a1e7beafe1b0d295
0112 20:51:33.594938 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.595045 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213735}
I20250112 20:51:33.595906 1185 tablet_replica.cc:331] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:33.596146 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 83c13126e53b46c0a1e7beafe1b0d295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c13126e53b46c0a1e7beafe1b0d295 already in progress: creating tablet
I20250112 20:51:33.596135 1185 raft_consensus.cc:2238] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:33.596527 1185 raft_consensus.cc:2267] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.597134 1185 ts_tablet_manager.cc:1905] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.597752 774 raft_consensus.cc:2463] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:33.598426 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 328) took 1221 ms. Trace:
I20250112 20:51:33.598701 1701 leader_election.cc:290] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:33.601828 1184 rpcz_store.cc:276] 0112 20:51:32.380880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.381246 (+ 366us) service_pool.cc:225] Handling call
0112 20:51:33.598265 (+1217019us) ts_tablet_manager.cc:1145] Deleting tablet 81b932b7e1e34d4ab18aae40047dc038
0112 20:51:33.598273 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.598375 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216796}
I20250112 20:51:33.602802 1178 raft_consensus.cc:357] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.603075 847 leader_election.cc:304] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:33.603365 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 81b932b7e1e34d4ab18aae40047dc038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81b932b7e1e34d4ab18aae40047dc038 already in progress: creating tablet
I20250112 20:51:33.604650 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72fda1743fc4f96b29add6e36e884de" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:33.604838 1178 raft_consensus.cc:383] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.604944 1178 raft_consensus.cc:738] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.605147 1704 raft_consensus.cc:2798] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.605414 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 330) took 1224 ms. Trace:
I20250112 20:51:33.605501 1184 rpcz_store.cc:276] 0112 20:51:32.381395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.381439 (+ 44us) service_pool.cc:225] Handling call
0112 20:51:33.605252 (+1223813us) ts_tablet_manager.cc:1145] Deleting tablet ac4e304a1212412f8e99bfd222a110b5
0112 20:51:33.605262 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.605367 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223635}
I20250112 20:51:33.605684 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f769533bfde0448792d01b883e41763c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:33.605926 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ac4e304a1212412f8e99bfd222a110b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4e304a1212412f8e99bfd222a110b5 already in progress: creating tablet
I20250112 20:51:33.606773 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 339) took 1173 ms. Trace:
I20250112 20:51:33.607034 1184 rpcz_store.cc:276] 0112 20:51:32.433632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.434114 (+ 482us) service_pool.cc:225] Handling call
0112 20:51:33.606618 (+1172504us) ts_tablet_manager.cc:1145] Deleting tablet 8d08048c402345a686d1e3fb3039bbc1
0112 20:51:33.606627 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.606726 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172306}
W20250112 20:51:33.607422 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56544c3b0cf140089baa678765a2eb79 consensus state has no leader
W20250112 20:51:33.607461 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8d08048c402345a686d1e3fb3039bbc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d08048c402345a686d1e3fb3039bbc1 already in progress: creating tablet
I20250112 20:51:33.607640 1704 raft_consensus.cc:695] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:33.606973 1163 tablet_bootstrap.cc:654] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.607944 1704 consensus_queue.cc:237] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.608747 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 340) took 1175 ms. Trace:
I20250112 20:51:33.607278 1178 consensus_queue.cc:260] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.608860 1184 rpcz_store.cc:276] 0112 20:51:32.433735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.433779 (+ 44us) service_pool.cc:225] Handling call
0112 20:51:33.608589 (+1174810us) ts_tablet_manager.cc:1145] Deleting tablet ad1a4b5ac2f94874b99b3ef790a034fd
0112 20:51:33.608598 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.608704 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164401}
W20250112 20:51:33.609261 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ad1a4b5ac2f94874b99b3ef790a034fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad1a4b5ac2f94874b99b3ef790a034fd already in progress: creating tablet
I20250112 20:51:33.610285 1178 ts_tablet_manager.cc:1428] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250112 20:51:33.611284 1184 tablet_replica.cc:331] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.611492 1184 raft_consensus.cc:2238] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.611603 910 raft_consensus.cc:2388] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:33.611687 1184 raft_consensus.cc:2267] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.616564 1780 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.617085 1761 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.620497 1164 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:33.622380 847 leader_election.cc:343] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.622963 1178 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:33.623107 983 leader_election.cc:304] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.623399 1821 raft_consensus.cc:3054] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.625042 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a48df2ec556d43a3b22129b1177ebf74
W20250112 20:51:33.625947 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b7b788acf8047acb34ca2596bbf141b consensus state has no leader
I20250112 20:51:33.626137 1821 raft_consensus.cc:2743] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.627988 1186 ts_tablet_manager.cc:1918] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.628131 1186 log.cc:1198] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/304e1c8abe094bd99a99fd358b697f81
I20250112 20:51:33.628268 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet a48df2ec556d43a3b22129b1177ebf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.628898 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a48df2ec556d43a3b22129b1177ebf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.629329 1178 tablet_bootstrap.cc:654] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.631443 1184 ts_tablet_manager.cc:1905] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.637745 1186 ts_tablet_manager.cc:1939] T 304e1c8abe094bd99a99fd358b697f81 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:33.638159 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 312) took 1340 ms. Trace:
I20250112 20:51:33.638356 1186 rpcz_store.cc:276] 0112 20:51:32.298087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.299567 (+ 1480us) service_pool.cc:225] Handling call
0112 20:51:33.576329 (+1276762us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:33.576338 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.600763 (+ 24425us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.627599 (+ 26836us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.638114 (+ 10515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275350}
I20250112 20:51:33.638928 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 325) took 1256 ms. Trace:
I20250112 20:51:33.638903 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 304e1c8abe094bd99a99fd358b697f81 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
I20250112 20:51:33.639077 1186 rpcz_store.cc:276] 0112 20:51:32.382038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.384296 (+ 2258us) service_pool.cc:225] Handling call
0112 20:51:33.638736 (+1254440us) ts_tablet_manager.cc:1145] Deleting tablet 1a04c223889f424e831e82a52a84f804
0112 20:51:33.638746 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.638888 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254228}
W20250112 20:51:33.639596 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1a04c223889f424e831e82a52a84f804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a04c223889f424e831e82a52a84f804 already in progress: creating tablet
I20250112 20:51:33.641474 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 327) took 1259 ms. Trace:
I20250112 20:51:33.641644 1186 rpcz_store.cc:276] 0112 20:51:32.382266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.384512 (+ 2246us) service_pool.cc:225] Handling call
0112 20:51:33.639361 (+1254849us) ts_tablet_manager.cc:1145] Deleting tablet 81b932b7e1e34d4ab18aae40047dc038
0112 20:51:33.639370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.641438 (+ 2068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254698}
I20250112 20:51:33.641978 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 328) took 1259 ms. Trace:
I20250112 20:51:33.642166 1163 tablet_bootstrap.cc:492] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.642315 1163 ts_tablet_manager.cc:1397] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.004s
W20250112 20:51:33.642331 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 81b932b7e1e34d4ab18aae40047dc038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81b932b7e1e34d4ab18aae40047dc038 already in progress: creating tablet
I20250112 20:51:33.643576 1164 ts_tablet_manager.cc:1397] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.136s user 0.002s sys 0.014s
I20250112 20:51:33.644623 1163 raft_consensus.cc:357] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.644881 1163 raft_consensus.cc:383] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.644954 1163 raft_consensus.cc:738] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.645217 1163 consensus_queue.cc:260] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.645848 1163 ts_tablet_manager.cc:1428] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:33.646124 1163 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.647251 1186 rpcz_store.cc:276] 0112 20:51:32.382391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.384702 (+ 2311us) service_pool.cc:225] Handling call
0112 20:51:33.641855 (+1257153us) ts_tablet_manager.cc:1145] Deleting tablet 83c13126e53b46c0a1e7beafe1b0d295
0112 20:51:33.641863 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.641937 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256916}
W20250112 20:51:33.648203 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 83c13126e53b46c0a1e7beafe1b0d295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c13126e53b46c0a1e7beafe1b0d295 already in progress: creating tablet
I20250112 20:51:33.650528 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 326) took 1266 ms. Trace:
I20250112 20:51:33.650709 1186 rpcz_store.cc:276] 0112 20:51:32.382142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.384835 (+ 2693us) service_pool.cc:225] Handling call
0112 20:51:33.647499 (+1262664us) ts_tablet_manager.cc:1145] Deleting tablet d0018e1eabf24241a358a7e00b6b5907
0112 20:51:33.647508 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.648747 (+ 1239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1262440}
I20250112 20:51:33.651026 1186 tablet_replica.cc:331] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:33.651510 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d0018e1eabf24241a358a7e00b6b5907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0018e1eabf24241a358a7e00b6b5907 already in progress: creating tablet
I20250112 20:51:33.652158 1821 raft_consensus.cc:491] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.652312 1821 raft_consensus.cc:513] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.652705 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a48df2ec556d43a3b22129b1177ebf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.652940 1821 leader_election.cc:290] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.653136 1164 raft_consensus.cc:357] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.653348 1164 raft_consensus.cc:383] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.653425 1164 raft_consensus.cc:738] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.653632 1825 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.653697 1164 consensus_queue.cc:260] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.654729 1164 ts_tablet_manager.cc:1428] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
W20250112 20:51:33.655341 983 leader_election.cc:343] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:33.655030 1164 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.657289 1164 tablet_bootstrap.cc:654] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.658630 1186 raft_consensus.cc:2238] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.658877 1186 raft_consensus.cc:2267] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.659713 1186 ts_tablet_manager.cc:1905] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.663937 1185 ts_tablet_manager.cc:1918] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.664168 1185 log.cc:1198] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/304e1c8abe094bd99a99fd358b697f81
I20250112 20:51:33.665026 1185 ts_tablet_manager.cc:1939] T 304e1c8abe094bd99a99fd358b697f81 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.665915 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 305) took 1806 ms. Trace:
I20250112 20:51:33.666079 1185 rpcz_store.cc:276] 0112 20:51:31.859353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.859519 (+ 166us) service_pool.cc:225] Handling call
0112 20:51:33.592288 (+1732769us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:33.592298 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.659862 (+ 67564us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.663827 (+ 3965us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.665876 (+ 2049us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732586}
I20250112 20:51:33.666631 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 304e1c8abe094bd99a99fd358b697f81 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
I20250112 20:51:33.668377 1178 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.668558 1178 ts_tablet_manager.cc:1397] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.004s
I20250112 20:51:33.668483 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d770e0399e80491f8de9b08d1eb2f139" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.669189 910 raft_consensus.cc:2463] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:33.669698 983 leader_election.cc:304] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.670121 1821 raft_consensus.cc:2798] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.670199 1821 raft_consensus.cc:491] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.670281 1821 raft_consensus.cc:3054] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.671933 1163 tablet_bootstrap.cc:654] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.673401 1821 raft_consensus.cc:513] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.675071 1821 leader_election.cc:290] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:33.675340 983 leader_election.cc:343] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:33.675786 1185 tablet_replica.cc:331] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:33.675801 1821 raft_consensus.cc:491] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.675971 1821 raft_consensus.cc:513] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.676294 1185 raft_consensus.cc:2238] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.676489 1185 raft_consensus.cc:2267] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.676816 1163 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.676936 1163 ts_tablet_manager.cc:1397] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20250112 20:51:33.677625 1163 raft_consensus.cc:357] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.677767 1185 ts_tablet_manager.cc:1905] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:33.678117 983 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4c3f359655c94f1d905943069548845b
I20250112 20:51:33.678251 1821 leader_election.cc:290] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.678740 1163 raft_consensus.cc:383] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.678447 1178 raft_consensus.cc:357] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.678964 1178 raft_consensus.cc:383] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.679000 1163 raft_consensus.cc:738] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.679127 1178 raft_consensus.cc:738] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.679371 1178 consensus_queue.cc:260] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.680050 1178 ts_tablet_manager.cc:1428] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:33.679419 1163 consensus_queue.cc:260] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.680351 1178 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:33.680958 1163 ts_tablet_manager.cc:1428] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:33.681712 1163 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.675195 1184 ts_tablet_manager.cc:1918] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.683856 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d770e0399e80491f8de9b08d1eb2f139" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.684640 910 raft_consensus.cc:3054] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.687554 910 raft_consensus.cc:2463] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:33.687916 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3f359655c94f1d905943069548845b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.689811 912 raft_consensus.cc:2463] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:33.689906 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f54731064e4948fdbca8cf87ae2a9533" } modify_external_catalogs: true
I20250112 20:51:33.690116 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f54731064e4948fdbca8cf87ae2a9533" } modify_external_catalogs: true
I20250112 20:51:33.690757 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f4382c24314948299813897344e2ae57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.691313 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4382c24314948299813897344e2ae57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.699134 1178 tablet_bootstrap.cc:654] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.699211 1164 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.699349 1164 ts_tablet_manager.cc:1397] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20250112 20:51:33.700917 1186 ts_tablet_manager.cc:1918] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.701120 1186 log.cc:1198] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:33.701740 1186 ts_tablet_manager.cc:1939] T ac4e304a1212412f8e99bfd222a110b5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:33.702860 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 329) took 1320 ms. Trace:
I20250112 20:51:33.711920 1164 raft_consensus.cc:357] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.712138 1164 raft_consensus.cc:383] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.712227 1164 raft_consensus.cc:738] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.713024 1164 consensus_queue.cc:260] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.712482 1704 consensus_queue.cc:1035] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.713788 1186 rpcz_store.cc:276] 0112 20:51:32.382517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.384969 (+ 2452us) service_pool.cc:225] Handling call
0112 20:51:33.650960 (+1265991us) ts_tablet_manager.cc:1145] Deleting tablet ac4e304a1212412f8e99bfd222a110b5
0112 20:51:33.650969 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.693478 (+ 42509us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.699010 (+ 5532us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.702797 (+ 3787us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1265815}
I20250112 20:51:33.714399 1186 tablet_replica.cc:331] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:33.714347 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ac4e304a1212412f8e99bfd222a110b5 (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:33.714665 1186 raft_consensus.cc:2238] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.715101 1186 raft_consensus.cc:2267] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.715886 1164 ts_tablet_manager.cc:1428] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250112 20:51:33.715937 1186 ts_tablet_manager.cc:1905] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.716233 1663 raft_consensus.cc:491] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.721138 1663 raft_consensus.cc:513] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.721750 1663 leader_election.cc:290] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.722146 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:33.722915 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ac72a7aedf80477ea4d02c7a2f150f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.723457 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac72a7aedf80477ea4d02c7a2f150f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.726099 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.726296 912 raft_consensus.cc:2463] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:33.727385 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.728008 710 leader_election.cc:304] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.728695 1663 raft_consensus.cc:2798] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.728710 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.728816 1663 raft_consensus.cc:491] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.728919 1663 raft_consensus.cc:3054] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.731614 1663 raft_consensus.cc:513] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.733315 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.733557 912 raft_consensus.cc:3054] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.737481 912 raft_consensus.cc:2463] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:33.738427 710 leader_election.cc:304] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.738727 1638 raft_consensus.cc:2798] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.738883 1638 raft_consensus.cc:695] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:33.739168 1638 consensus_queue.cc:237] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.743060 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e9f669a739d94f99a5b7e9ccccd29545
I20250112 20:51:33.744735 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.744798 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 54ab0ae10f09469fa4f913e049002481 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.745595 880 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.746312 1857 consensus_queue.cc:1035] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.747511 1663 leader_election.cc:290] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.752307 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_343ab21a9b504bef9d7ceb9a38d3870c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.752995 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_343ab21a9b504bef9d7ceb9a38d3870c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.752934 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c8ca9356c6d94a34b8bca0f720df2028"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.753453 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8ca9356c6d94a34b8bca0f720df2028 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.755290 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f34aa32d6474561988b2cf3d0e8fd3b from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.755602 1164 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.755930 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fd0dbb26e1be41e1a0aa881aa508c9a1" } new_table_name: "default.Table_8cb2067a639c442685fda6b357d895f6" modify_external_catalogs: true
I20250112 20:51:33.757819 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 4561956c97824824a55283a8da21929e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.683804 1163 tablet_bootstrap.cc:654] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.688733 983 leader_election.cc:304] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.763031 1821 raft_consensus.cc:2798] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.763166 1821 raft_consensus.cc:695] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
W20250112 20:51:33.763135 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 74399.5us
I20250112 20:51:33.763454 1821 consensus_queue.cc:237] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.763589 983 leader_election.cc:304] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.727919 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.764441 1824 raft_consensus.cc:2798] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.764539 1824 raft_consensus.cc:491] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.764629 1824 raft_consensus.cc:3054] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.732580 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:33.746551 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:33.767571 710 leader_election.cc:343] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.683705 1184 log.cc:1198] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8c38c8594a384689ae57c26174765284
I20250112 20:51:33.728070 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.781458 1178 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.781626 1178 ts_tablet_manager.cc:1397] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.010s
W20250112 20:51:33.782289 710 leader_election.cc:343] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.783031 1164 tablet_bootstrap.cc:654] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.787241 1186 ts_tablet_manager.cc:1918] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.787375 1186 log.cc:1198] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f769533bfde0448792d01b883e41763c
I20250112 20:51:33.787824 1186 ts_tablet_manager.cc:1939] T f769533bfde0448792d01b883e41763c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:33.787317 1824 raft_consensus.cc:513] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.788724 1824 leader_election.cc:290] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.788960 1163 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.788941 1178 raft_consensus.cc:357] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.789120 1163 ts_tablet_manager.cc:1397] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.107s user 0.001s sys 0.003s
I20250112 20:51:33.789140 1178 raft_consensus.cc:383] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.789209 1178 raft_consensus.cc:738] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.789417 1178 consensus_queue.cc:260] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.789803 1163 raft_consensus.cc:357] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.790009 1163 raft_consensus.cc:383] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.790086 1163 raft_consensus.cc:738] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.790273 1163 consensus_queue.cc:260] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.790767 1163 ts_tablet_manager.cc:1428] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:33.791383 1163 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.793414 1178 ts_tablet_manager.cc:1428] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:33.793741 1178 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:33.794062 1857 consensus_queue.cc:1035] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.800424 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3f359655c94f1d905943069548845b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.800623 912 raft_consensus.cc:3054] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.802459 1184 ts_tablet_manager.cc:1939] T 8c38c8594a384689ae57c26174765284 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.802770 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 2469127891524be5ac3dc13fbcf35c7a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.803400 1163 tablet_bootstrap.cc:654] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.803998 912 raft_consensus.cc:2463] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:33.805687 983 leader_election.cc:304] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:33.805758 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1565c7487c428f8f963aca406263fe (DEFAULT_TABLE table=default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.806172 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1565c7487c428f8f963aca406263fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.806216 1824 raft_consensus.cc:2798] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.806360 1824 raft_consensus.cc:695] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:33.806615 1824 consensus_queue.cc:237] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.809545 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:33.809815 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1565c7487c428f8f963aca406263fe (DEFAULT_TABLE table=default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.809854 453 catalog_manager.cc:5526] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:33.810212 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1565c7487c428f8f963aca406263fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:33.811286 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1565c7487c428f8f963aca406263fe (DEFAULT_TABLE table=default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:33.811652 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1565c7487c428f8f963aca406263fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:33.817548 983 leader_election.cc:343] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4c3f359655c94f1d905943069548845b
I20250112 20:51:33.818275 1164 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.819224 1164 ts_tablet_manager.cc:1397] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
W20250112 20:51:33.821019 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c11a892641554e4db9bdd558068cac40 consensus state has no leader
I20250112 20:51:33.823273 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_626c18b182014d509b9de95803204412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.823866 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_626c18b182014d509b9de95803204412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.827993 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0c9b5c2846724b71a49b34852f5bf4aa" } new_table_name: "default.Table_6476b4820a6f465f9b721e9bd7b463d2" modify_external_catalogs: true
I20250112 20:51:33.788740 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 330) took 1406 ms. Trace:
I20250112 20:51:33.831072 1186 rpcz_store.cc:276] 0112 20:51:32.382625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.387057 (+ 4432us) service_pool.cc:225] Handling call
0112 20:51:33.714316 (+1327259us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:33.714329 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.719783 (+ 5454us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.787110 (+ 67327us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.788694 (+ 1584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1327058}
I20250112 20:51:33.831434 1186 tablet_replica.cc:331] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:33.831666 1186 raft_consensus.cc:2238] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.832020 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4246de22bb514400b9eb57117e36a6f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.832573 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4246de22bb514400b9eb57117e36a6f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.834447 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f769533bfde0448792d01b883e41763c (table default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]) successfully deleted
I20250112 20:51:33.835079 1048 raft_consensus.cc:3054] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.840083 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8c38c8594a384689ae57c26174765284 (table default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]) successfully deleted
I20250112 20:51:33.840646 1184 tablet_replica.cc:331] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.840907 1184 raft_consensus.cc:2238] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.841101 1184 raft_consensus.cc:2267] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.843274 1163 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.843422 1163 ts_tablet_manager.cc:1397] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20250112 20:51:33.844172 1163 raft_consensus.cc:357] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.844391 1163 raft_consensus.cc:383] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.844485 1163 raft_consensus.cc:738] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.844705 1163 consensus_queue.cc:260] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.844856 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:33.845640 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_70e7a6a54ee6434988d6cfb21ade1f02" } modify_external_catalogs: true
I20250112 20:51:33.845813 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_70e7a6a54ee6434988d6cfb21ade1f02" } modify_external_catalogs: true
I20250112 20:51:33.845856 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.846153 1163 ts_tablet_manager.cc:1428] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:33.846292 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.846709 1164 raft_consensus.cc:357] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.846953 1164 raft_consensus.cc:383] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.847162 1163 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.848771 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e]). Expected version 1 got 0
I20250112 20:51:33.849252 1163 tablet_bootstrap.cc:654] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.849659 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4] alter complete (version 1)
I20250112 20:51:33.849817 1185 ts_tablet_manager.cc:1918] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.849964 1185 log.cc:1198] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8c38c8594a384689ae57c26174765284
I20250112 20:51:33.850374 1185 ts_tablet_manager.cc:1939] T 8c38c8594a384689ae57c26174765284 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.850730 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.850802 1184 ts_tablet_manager.cc:1905] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.850965 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 6fe28f23448c47d3979f1982f939e39c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:33.850996 1164 raft_consensus.cc:738] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.851523 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_bb77de2d74804434a45865840ea7eea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.852048 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb77de2d74804434a45865840ea7eea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.854771 1164 consensus_queue.cc:260] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.855418 1164 ts_tablet_manager.cc:1428] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250112 20:51:33.855726 1164 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:33.857409 1164 tablet_bootstrap.cc:654] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.857882 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4561956c97824824a55283a8da21929e
I20250112 20:51:33.858440 454 catalog_manager.cc:6596] ReplaceTablet: tablet 4561956c97824824a55283a8da21929e deleted and replaced by tablet d80791d8582446b89adf212d3d6507eb
I20250112 20:51:33.858775 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:33.859226 447 catalog_manager.cc:6596] ReplaceTablet: tablet 2469127891524be5ac3dc13fbcf35c7a deleted and replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62
I20250112 20:51:33.861398 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:33.863948 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5ea7336d13ea46748167e35e7e7e8c98 (table default.Table_e5db37687d384be894f7f6239198cd82 [id=11743480162f4442b00dc54a796e3461], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5ea7336d13ea46748167e35e7e7e8c98 consensus state has no leader
I20250112 20:51:33.863951 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.864230 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:33.864737 452 catalog_manager.cc:6596] ReplaceTablet: tablet 4f34aa32d6474561988b2cf3d0e8fd3b deleted and replaced by tablet 549d92191a594788bf639cd60a6b5142
I20250112 20:51:33.865514 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.867424 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:33.858119 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 306) took 1998 ms. Trace:
I20250112 20:51:33.859229 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.859944 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:33.872503 1185 rpcz_store.cc:276] 0112 20:51:31.859843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.860036 (+ 193us) service_pool.cc:225] Handling call
0112 20:51:33.675717 (+1815681us) ts_tablet_manager.cc:1145] Deleting tablet 8c38c8594a384689ae57c26174765284
0112 20:51:33.675726 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.785490 (+109764us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.849668 (+ 64178us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.858066 (+ 8398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815497}
I20250112 20:51:33.873677 1184 ts_tablet_manager.cc:1918] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.875406 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8c38c8594a384689ae57c26174765284 (table default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]) successfully deleted
I20250112 20:51:33.875828 1185 tablet_replica.cc:331] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:33.876173 1185 raft_consensus.cc:2238] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:33.876328 1163 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.876571 1163 ts_tablet_manager.cc:1397] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.004s
I20250112 20:51:33.876646 1185 raft_consensus.cc:2267] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.877296 1185 ts_tablet_manager.cc:1905] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.877430 1163 raft_consensus.cc:357] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.877707 1163 raft_consensus.cc:383] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.877820 1163 raft_consensus.cc:738] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.878121 1163 consensus_queue.cc:260] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.880085 1163 ts_tablet_manager.cc:1428] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:33.880570 1163 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.884891 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_284c5bfc11f1458dbade76ac78b87f20" } new_table_name: "default.Table_abecc332f4f147418a2479c3e0f768f5" modify_external_catalogs: true
I20250112 20:51:33.886193 1184 log.cc:1198] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/6f4421d16e3f423794c44970a2469b10
I20250112 20:51:33.886693 1184 ts_tablet_manager.cc:1939] T 6f4421d16e3f423794c44970a2469b10 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.888442 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b2428906915d47bd9fb06b88e971194c" } new_table_name: "default.Table_e0fd0e3995274cd9b07af514ac3ef3cd" modify_external_catalogs: true
I20250112 20:51:33.889807 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 6f4421d16e3f423794c44970a2469b10 (table default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]) successfully deleted
I20250112 20:51:33.889994 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ebead212ebdd45e29399da06bdace3c8" } modify_external_catalogs: true
I20250112 20:51:33.890156 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ebead212ebdd45e29399da06bdace3c8" } modify_external_catalogs: true
I20250112 20:51:33.890473 1163 tablet_bootstrap.cc:654] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.891464 1184 tablet_replica.cc:331] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.891754 1184 raft_consensus.cc:2238] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.892530 1185 ts_tablet_manager.cc:1918] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:33.892714 1184 raft_consensus.cc:2267] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.894124 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c3afc9f0dcf0422fa7cdd946feec0e58" } new_table_name: "default.Table_80d55cc666794ca8853848350b2cb0e6" modify_external_catalogs: true
I20250112 20:51:33.896888 1824 raft_consensus.cc:491] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.897086 1824 raft_consensus.cc:513] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.897851 1824 leader_election.cc:290] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.896883 1184 ts_tablet_manager.cc:1905] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.899806 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.904367 774 raft_consensus.cc:2463] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:33.904891 1185 log.cc:1198] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/6f4421d16e3f423794c44970a2469b10
I20250112 20:51:33.904892 983 leader_election.cc:304] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.905233 1824 raft_consensus.cc:2798] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.905359 1824 raft_consensus.cc:491] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.905438 1824 raft_consensus.cc:3054] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.905867 1185 ts_tablet_manager.cc:1939] T 6f4421d16e3f423794c44970a2469b10 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:33.907137 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 307) took 2046 ms. Trace:
I20250112 20:51:33.908102 1824 raft_consensus.cc:513] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.908200 1185 rpcz_store.cc:276] 0112 20:51:31.860368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.860554 (+ 186us) service_pool.cc:225] Handling call
0112 20:51:33.875724 (+2015170us) ts_tablet_manager.cc:1145] Deleting tablet 6f4421d16e3f423794c44970a2469b10
0112 20:51:33.875737 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.889499 (+ 13762us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.892405 (+ 2906us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.907100 (+ 14695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014966}
I20250112 20:51:33.909051 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 6f4421d16e3f423794c44970a2469b10 (table default.Table_fb52d5971de44b71b7a53e1d05798b74 [id=ae104cf7ff884bbf91229fc6c2a6d277]) successfully deleted
I20250112 20:51:33.909333 1824 leader_election.cc:290] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.909873 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:33.910118 774 raft_consensus.cc:3054] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.913347 774 raft_consensus.cc:2463] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:33.913769 983 leader_election.cc:304] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.914119 1824 raft_consensus.cc:2798] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:33.914290 1824 raft_consensus.cc:695] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:33.914810 1824 consensus_queue.cc:237] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.919533 1163 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.919703 1163 ts_tablet_manager.cc:1397] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.005s
I20250112 20:51:33.920428 1163 raft_consensus.cc:357] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.920604 1163 raft_consensus.cc:383] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.920678 1163 raft_consensus.cc:738] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.838948 1704 consensus_queue.cc:1035] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.846098 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.848434 1178 tablet_bootstrap.cc:654] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.860296 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.861006 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.865316 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:33.898598 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.921685 1163 consensus_queue.cc:260] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.922293 1163 ts_tablet_manager.cc:1428] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:33.922596 1163 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.909364 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.917929 1701 raft_consensus.cc:491] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.923270 1701 raft_consensus.cc:513] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.924078 1701 leader_election.cc:290] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:33.924327 1163 tablet_bootstrap.cc:654] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.924796 912 raft_consensus.cc:2463] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:33.926429 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:33.926693 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ee62fff7d4cf7b4aaeacf63ed8e02" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:33.927065 770 raft_consensus.cc:2388] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:33.928488 847 leader_election.cc:304] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:33.928833 1858 raft_consensus.cc:3054] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.929373 1663 consensus_queue.cc:1035] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:33.932257 1858 raft_consensus.cc:2743] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:33.933224 1186 raft_consensus.cc:2267] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.933907 1186 ts_tablet_manager.cc:1905] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.939426 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f89fdf37a8944ee0be9e90eec745a613" } modify_external_catalogs: true
I20250112 20:51:33.939662 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f89fdf37a8944ee0be9e90eec745a613" } modify_external_catalogs: true
I20250112 20:51:33.944151 1184 ts_tablet_manager.cc:1918] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:33.945047 1184 log.cc:1198] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/0e2235e2a63c4e1192b3260085908445
I20250112 20:51:33.945578 1184 ts_tablet_manager.cc:1939] T 0e2235e2a63c4e1192b3260085908445 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:33.947187 1184 tablet_replica.cc:331] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.947185 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 0e2235e2a63c4e1192b3260085908445 (table default.Table_e3b1f099596246f78012a56ae87a63bd [id=912f6096dcc441cd89998179f8bd86c1]) successfully deleted
W20250112 20:51:33.947492 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d570daec845f40778c48ff3f2e2e30cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d570daec845f40778c48ff3f2e2e30cb already in progress: creating tablet
I20250112 20:51:33.950474 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_25113f809e324283a5b53261123398f1" } modify_external_catalogs: true
I20250112 20:51:33.950736 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_25113f809e324283a5b53261123398f1" } modify_external_catalogs: true
I20250112 20:51:33.952387 1184 raft_consensus.cc:2238] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.952982 1184 raft_consensus.cc:2267] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.954123 1184 ts_tablet_manager.cc:1905] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.957029 1825 tablet.cc:1721] T 1c31c29073aa46ffa847905be1595a78 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:33.958524 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e72a6c8e18934b789a048d5aaded121b consensus state has no leader
I20250112 20:51:33.959528 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fd99ee08b68344f499dba32ccf27ff4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.960072 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd99ee08b68344f499dba32ccf27ff4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.961935 1163 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.962157 1163 ts_tablet_manager.cc:1397] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.012s
I20250112 20:51:33.962905 1163 raft_consensus.cc:357] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.963130 1163 raft_consensus.cc:383] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.963214 1163 raft_consensus.cc:738] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.963549 1163 consensus_queue.cc:260] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.964381 1163 ts_tablet_manager.cc:1428] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:33.964660 1163 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.966737 1163 tablet_bootstrap.cc:654] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:33.968861 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4ef01b41b6140d6bdad07bc79dd7dc5 consensus state has no leader
I20250112 20:51:33.969890 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cdede98843a14d8e83eb0d3a74827353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:33.970456 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdede98843a14d8e83eb0d3a74827353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.973158 1184 ts_tablet_manager.cc:1918] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:33.973799 1184 log.cc:1198] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/44f11304909b416aa1e9db1abf633a89
I20250112 20:51:33.974355 1184 ts_tablet_manager.cc:1939] T 44f11304909b416aa1e9db1abf633a89 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:33.978441 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22f6b2c4ea5f4167896c78b1c64cbb62 has no consensus state
I20250112 20:51:33.979033 1780 tablet.cc:1721] T 1c31c29073aa46ffa847905be1595a78 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.980464 1178 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:33.980473 1164 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:33.980612 1178 ts_tablet_manager.cc:1397] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.006s
I20250112 20:51:33.980751 1164 ts_tablet_manager.cc:1397] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.007s
I20250112 20:51:33.981017 1761 tablet.cc:1721] T 1c31c29073aa46ffa847905be1595a78 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:33.981420 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 308) took 2119 ms. Trace:
I20250112 20:51:33.981678 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 370) took 1023 ms. Trace:
I20250112 20:51:33.981662 1164 raft_consensus.cc:357] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.982187 1164 raft_consensus.cc:383] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.982276 1164 raft_consensus.cc:738] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.982364 1178 raft_consensus.cc:357] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.982785 1178 raft_consensus.cc:383] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.982877 1178 raft_consensus.cc:738] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:33.982785 1164 consensus_queue.cc:260] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.983083 1178 consensus_queue.cc:260] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:33.985024 1163 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:33.985163 1163 ts_tablet_manager.cc:1397] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20250112 20:51:33.985888 1163 raft_consensus.cc:357] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.985881 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b5d6bc8b522a4be197bd045c77d4a5f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:33.986099 1163 raft_consensus.cc:383] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:33.986178 1163 raft_consensus.cc:738] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:33.986428 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5d6bc8b522a4be197bd045c77d4a5f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:33.986374 1163 consensus_queue.cc:260] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.987112 1163 ts_tablet_manager.cc:1428] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:33.987277 1178 ts_tablet_manager.cc:1428] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250112 20:51:33.987533 1163 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:33.981767 1184 rpcz_store.cc:276] 0112 20:51:32.958134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.958269 (+ 135us) service_pool.cc:225] Handling call
0112 20:51:33.947120 (+988851us) ts_tablet_manager.cc:1145] Deleting tablet 44f11304909b416aa1e9db1abf633a89
0112 20:51:33.947127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.958066 (+ 10939us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.973046 (+ 14980us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:33.981617 (+ 8571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988703}
I20250112 20:51:33.988334 1178 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:33.988710 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 371) took 1030 ms. Trace:
I20250112 20:51:33.988843 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]) successfully deleted
I20250112 20:51:33.988968 1663 raft_consensus.cc:491] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:33.989120 1663 raft_consensus.cc:513] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.989769 1663 leader_election.cc:290] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:33.989918 1184 rpcz_store.cc:276] 0112 20:51:32.958242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.958438 (+ 196us) service_pool.cc:225] Handling call
0112 20:51:33.988531 (+1030093us) ts_tablet_manager.cc:1145] Deleting tablet 7deb6e37f75e4aebacb04c4553049940
0112 20:51:33.988539 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.988645 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029891}
I20250112 20:51:33.990373 1178 tablet_bootstrap.cc:654] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.990399 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 369) took 1032 ms. Trace:
I20250112 20:51:33.990988 1184 rpcz_store.cc:276] 0112 20:51:32.958024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.959287 (+ 1263us) service_pool.cc:225] Handling call
0112 20:51:33.990234 (+1030947us) ts_tablet_manager.cc:1145] Deleting tablet 42eb0f79df5444f8a8360921d5f2b9ee
0112 20:51:33.990242 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.990345 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030778}
I20250112 20:51:33.991269 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:33.991392 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 373) took 1032 ms. Trace:
I20250112 20:51:33.991518 912 raft_consensus.cc:2463] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:33.991466 1184 rpcz_store.cc:276] 0112 20:51:32.958458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.959472 (+ 1014us) service_pool.cc:225] Handling call
0112 20:51:33.991255 (+1031783us) ts_tablet_manager.cc:1145] Deleting tablet 8f636df32bd34438bb54989736eaa431
0112 20:51:33.991265 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.991345 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031603}
I20250112 20:51:33.991757 1184 tablet_replica.cc:331] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:33.991986 710 leader_election.cc:304] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:33.990907 1163 tablet_bootstrap.cc:654] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:33.992281 1663 raft_consensus.cc:2798] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:33.992426 1663 raft_consensus.cc:491] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:33.992525 1663 raft_consensus.cc:3054] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:33.992020 1184 raft_consensus.cc:2238] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:33.993206 1184 raft_consensus.cc:2267] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:33.993688 1184 ts_tablet_manager.cc:1905] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:33.995102 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:33.995601 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7deb6e37f75e4aebacb04c4553049940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7deb6e37f75e4aebacb04c4553049940 already in progress: creating tablet
W20250112 20:51:33.995851 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 42eb0f79df5444f8a8360921d5f2b9ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42eb0f79df5444f8a8360921d5f2b9ee already in progress: creating tablet
W20250112 20:51:33.996071 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8f636df32bd34438bb54989736eaa431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f636df32bd34438bb54989736eaa431 already in progress: creating tablet
I20250112 20:51:33.997241 1663 raft_consensus.cc:513] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.997959 1663 leader_election.cc:290] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:33.998286 710 leader_election.cc:343] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:33.999737 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:33.999974 912 raft_consensus.cc:3054] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.000648 1178 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.000841 1178 ts_tablet_manager.cc:1397] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20250112 20:51:34.002110 1178 raft_consensus.cc:357] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.002351 1178 raft_consensus.cc:383] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.002437 1178 raft_consensus.cc:738] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.005159 748 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e150c031e4f4e99966c14e9005d00 (DEFAULT_TABLE table=default.Table_f4382c24314948299813897344e2ae57 [id=d15fcac9ba9148c3a21be17a67a44ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.005576 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e150c031e4f4e99966c14e9005d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.006832 1184 ts_tablet_manager.cc:1918] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:34.006969 1184 log.cc:1198] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3ade9b410d8443249730cacdf7266eb2
I20250112 20:51:34.007336 1184 ts_tablet_manager.cc:1939] T 3ade9b410d8443249730cacdf7266eb2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.008102 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 358) took 1051 ms. Trace:
I20250112 20:51:34.008206 1184 rpcz_store.cc:276] 0112 20:51:32.956705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.960498 (+ 3793us) service_pool.cc:225] Handling call
0112 20:51:33.991671 (+1031173us) ts_tablet_manager.cc:1145] Deleting tablet 3ade9b410d8443249730cacdf7266eb2
0112 20:51:33.991680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.997201 (+ 5521us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.006705 (+ 9504us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.008068 (+ 1363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030987}
I20250112 20:51:34.008494 1184 tablet_replica.cc:331] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.008692 1184 raft_consensus.cc:2238] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.009685 886 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e150c031e4f4e99966c14e9005d00 (DEFAULT_TABLE table=default.Table_f4382c24314948299813897344e2ae57 [id=d15fcac9ba9148c3a21be17a67a44ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.010221 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e150c031e4f4e99966c14e9005d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.012833 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281] alter complete (version 1)
I20250112 20:51:34.013702 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_d8aa49cfb72d47f889b7646990bdaece [id=5e31685a4caa41729b85ec6b230333f6] alter complete (version 3)
I20250112 20:51:34.019361 1178 consensus_queue.cc:260] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.021390 1184 raft_consensus.cc:2267] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.021502 1178 ts_tablet_manager.cc:1428] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250112 20:51:34.021533 877 tablet_service.cc:1467] Processing CreateTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.021966 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f264d84ae4846d1afaaabc41d5789f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.022185 878 tablet_service.cc:1467] Processing CreateTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.022390 1184 ts_tablet_manager.cc:1905] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.022565 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.022742 1178 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.024036 882 tablet_service.cc:1467] Processing CreateTablet for tablet be4a237cfeeb4d0fb7ad1abb4ff97845 (DEFAULT_TABLE table=default.Table_0d195bfd940c4601bf584ba32f274f76 [id=5ed2e8826c644775aad86c77003248ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.024747 1178 tablet_bootstrap.cc:654] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.024897 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4a237cfeeb4d0fb7ad1abb4ff97845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.025106 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7972856872e7411e9a3651b4413ac951 (DEFAULT_TABLE table=default.Table_61b014e293d64e85a96563251b9229f6 [id=a632752018f9467d878ca829b750e19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.025496 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7972856872e7411e9a3651b4413ac951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.027252 1858 raft_consensus.cc:491] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.027408 1858 raft_consensus.cc:513] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.028041 1858 leader_election.cc:290] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.029083 880 tablet_service.cc:1467] Processing CreateTablet for tablet dde64da3dd084a4785986c091852cbc6 (DEFAULT_TABLE table=default.Table_1f79972c8942460f988e2d87f374fc5e [id=97c3a7bf3d874bec9dba134fb054d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.029429 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde64da3dd084a4785986c091852cbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.033170 888 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 (DEFAULT_TABLE table=default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e [id=c0f4f9a776794374b5cfc063becc2876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.033645 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fd4177f9394a5a82d4f5e02369b3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.033910 1178 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.034050 1178 ts_tablet_manager.cc:1397] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20250112 20:51:34.034814 1178 raft_consensus.cc:357] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.035029 1178 raft_consensus.cc:383] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.035133 1178 raft_consensus.cc:738] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.035430 1178 consensus_queue.cc:260] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:33.981554 1185 rpcz_store.cc:276] 0112 20:51:31.861325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:31.861623 (+ 298us) service_pool.cc:225] Handling call
0112 20:51:33.981009 (+2119386us) ts_tablet_manager.cc:1145] Deleting tablet 6092a31de9bc47e4bae5f5aebb180c6d
0112 20:51:33.981022 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:33.981151 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2117988}
I20250112 20:51:34.036168 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 309) took 1842 ms. Trace:
I20250112 20:51:34.036320 1185 rpcz_store.cc:276] 0112 20:51:32.193391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.193445 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:34.035922 (+1842477us) ts_tablet_manager.cc:1145] Deleting tablet 48e390eb2e404250a8b6edba2ee4f6d6
0112 20:51:34.035934 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.036113 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830700}
I20250112 20:51:34.036764 1185 tablet_replica.cc:331] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:34.037012 1178 ts_tablet_manager.cc:1428] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:34.037108 1185 raft_consensus.cc:2238] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.037128 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb (DEFAULT_TABLE table=default.Table_f9f2cdcb247848539893c00bfbef8b18 [id=a40bd70a54ec4854a36809ac58df29b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.037416 1185 raft_consensus.cc:2267] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.037596 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8fc4a45fe448b919ac001a7980deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.038100 1178 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.038301 1185 ts_tablet_manager.cc:1905] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.039738 1178 tablet_bootstrap.cc:654] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.041050 879 tablet_service.cc:1467] Processing CreateTablet for tablet f875f075e98e45499291b519caf03184 (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.041393 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f875f075e98e45499291b519caf03184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.044718 1184 ts_tablet_manager.cc:1918] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.044886 1184 log.cc:1198] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:34.045099 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1e584b761d234c5aa0ab875acd8e1e05 (DEFAULT_TABLE table=default.Table_09bd1e992b194beb825bb3280677c608 [id=27536d1722784720b208bbc108bdc02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.045291 1184 ts_tablet_manager.cc:1939] T bc5af0bc83bc415b972cbbe0fb39f75f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.045447 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e584b761d234c5aa0ab875acd8e1e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.046123 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 359) took 1089 ms. Trace:
I20250112 20:51:34.046211 1184 rpcz_store.cc:276] 0112 20:51:32.956835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.960712 (+ 3877us) service_pool.cc:225] Handling call
0112 20:51:34.008442 (+1047730us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:34.008450 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.029782 (+ 21332us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.044581 (+ 14799us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.046069 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047531}
I20250112 20:51:34.046538 1184 tablet_replica.cc:331] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.046725 1184 raft_consensus.cc:2238] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.046888 1184 pending_rounds.cc:62] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:34.046934 1184 pending_rounds.cc:69] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585017032568832 op_type: NO_OP noop_request { }
I20250112 20:51:34.047066 1184 raft_consensus.cc:2883] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:34.047160 1184 raft_consensus.cc:2267] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.048874 893 tablet_service.cc:1467] Processing CreateTablet for tablet b85b9e82f9e847c4acfeecb82201e694 (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.049306 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85b9e82f9e847c4acfeecb82201e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.051201 1178 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.051360 1178 ts_tablet_manager.cc:1397] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20250112 20:51:34.052744 1178 raft_consensus.cc:357] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.053014 1178 raft_consensus.cc:383] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.053103 884 tablet_service.cc:1467] Processing CreateTablet for tablet 104d6b4effb84314a6b2b9db883b4366 (DEFAULT_TABLE table=default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.053460 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d6b4effb84314a6b2b9db883b4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.054764 1184 ts_tablet_manager.cc:1905] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.053097 1178 raft_consensus.cc:738] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.056471 1178 consensus_queue.cc:260] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.056667 876 tablet_service.cc:1467] Processing CreateTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (DEFAULT_TABLE table=default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.056915 889 tablet_service.cc:1467] Processing CreateTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (DEFAULT_TABLE table=default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.057034 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.057229 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.058095 1178 ts_tablet_manager.cc:1428] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250112 20:51:34.059394 1178 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.061043 1178 tablet_bootstrap.cc:654] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.061890 1858 raft_consensus.cc:491] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.062093 1858 raft_consensus.cc:513] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.062754 1858 leader_election.cc:290] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:34.065771 874 tablet_service.cc:1467] Processing CreateTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 (DEFAULT_TABLE table=default.Table_c8ca9356c6d94a34b8bca0f720df2028 [id=2049ad7ffcee4affa56490ef29e949da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.065994 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84624dc685b401192ca753ff59da016" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.066264 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bea4d83ea644a0930a29f61e7a5fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.072364 1164 ts_tablet_manager.cc:1428] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20250112 20:51:34.072728 1184 ts_tablet_manager.cc:1918] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.072930 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f0875e271d4aaf9dc5da6aa8113ed0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.072945 1184 log.cc:1198] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/31d3669355bc4b6bbc665edaa80c7a9e
I20250112 20:51:34.073565 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2ce57ea2a0d44a2d817812e80fe5d573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.073951 1184 ts_tablet_manager.cc:1939] T 31d3669355bc4b6bbc665edaa80c7a9e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:34.074138 847 leader_election.cc:343] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:34.074311 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ce57ea2a0d44a2d817812e80fe5d573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:34.074375 847 leader_election.cc:343] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.075157 1178 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.075297 1178 ts_tablet_manager.cc:1397] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20250112 20:51:34.076066 1178 raft_consensus.cc:357] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.076263 1178 raft_consensus.cc:383] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.076356 1178 raft_consensus.cc:738] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.076571 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb (DEFAULT_TABLE table=default.Table_f9f2cdcb247848539893c00bfbef8b18 [id=a40bd70a54ec4854a36809ac58df29b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.076591 1178 consensus_queue.cc:260] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.077066 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8fc4a45fe448b919ac001a7980deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.077139 1178 ts_tablet_manager.cc:1428] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:34.077116 739 tablet_service.cc:1467] Processing CreateTablet for tablet f875f075e98e45499291b519caf03184 (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.077265 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.077481 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f875f075e98e45499291b519caf03184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.077639 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.078006 755 tablet_service.cc:1467] Processing CreateTablet for tablet 1e584b761d234c5aa0ab875acd8e1e05 (DEFAULT_TABLE table=default.Table_09bd1e992b194beb825bb3280677c608 [id=27536d1722784720b208bbc108bdc02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.078333 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e584b761d234c5aa0ab875acd8e1e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.078377 1178 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.080000 1178 tablet_bootstrap.cc:654] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.081035 754 tablet_service.cc:1467] Processing CreateTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.081239 752 tablet_service.cc:1467] Processing CreateTablet for tablet dde64da3dd084a4785986c091852cbc6 (DEFAULT_TABLE table=default.Table_1f79972c8942460f988e2d87f374fc5e [id=97c3a7bf3d874bec9dba134fb054d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.081411 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f264d84ae4846d1afaaabc41d5789f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.081656 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde64da3dd084a4785986c091852cbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.081890 737 tablet_service.cc:1467] Processing CreateTablet for tablet b85b9e82f9e847c4acfeecb82201e694 (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.082217 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85b9e82f9e847c4acfeecb82201e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.082921 1178 tablet_bootstrap.cc:492] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.083078 1178 ts_tablet_manager.cc:1397] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:34.084406 1178 raft_consensus.cc:357] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.084415 1164 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.084618 1178 raft_consensus.cc:383] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.084684 1178 raft_consensus.cc:738] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.084985 1178 consensus_queue.cc:260] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.085053 749 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 (DEFAULT_TABLE table=default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e [id=c0f4f9a776794374b5cfc063becc2876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.085244 753 tablet_service.cc:1467] Processing CreateTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (DEFAULT_TABLE table=default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.085420 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fd4177f9394a5a82d4f5e02369b3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.085515 1178 ts_tablet_manager.cc:1428] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:34.085695 747 tablet_service.cc:1467] Processing CreateTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (DEFAULT_TABLE table=default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.086043 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.086689 1178 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.088416 1178 tablet_bootstrap.cc:654] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.088769 756 tablet_service.cc:1467] Processing CreateTablet for tablet be4a237cfeeb4d0fb7ad1abb4ff97845 (DEFAULT_TABLE table=default.Table_0d195bfd940c4601bf584ba32f274f76 [id=5ed2e8826c644775aad86c77003248ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.081032 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7972856872e7411e9a3651b4413ac951 (DEFAULT_TABLE table=default.Table_61b014e293d64e85a96563251b9229f6 [id=a632752018f9467d878ca829b750e19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.089244 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4a237cfeeb4d0fb7ad1abb4ff97845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.089658 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.093050 738 tablet_service.cc:1467] Processing CreateTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 (DEFAULT_TABLE table=default.Table_c8ca9356c6d94a34b8bca0f720df2028 [id=2049ad7ffcee4affa56490ef29e949da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.093389 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bea4d83ea644a0930a29f61e7a5fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.093329 742 tablet_service.cc:1467] Processing CreateTablet for tablet 104d6b4effb84314a6b2b9db883b4366 (DEFAULT_TABLE table=default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.093641 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d6b4effb84314a6b2b9db883b4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.093850 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7972856872e7411e9a3651b4413ac951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.098522 1663 raft_consensus.cc:491] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.098781 1663 raft_consensus.cc:513] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.101581 1029 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e150c031e4f4e99966c14e9005d00 (DEFAULT_TABLE table=default.Table_f4382c24314948299813897344e2ae57 [id=d15fcac9ba9148c3a21be17a67a44ae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.101768 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 7972856872e7411e9a3651b4413ac951 (DEFAULT_TABLE table=default.Table_61b014e293d64e85a96563251b9229f6 [id=a632752018f9467d878ca829b750e19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.102031 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7e150c031e4f4e99966c14e9005d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.102244 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7972856872e7411e9a3651b4413ac951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.105116 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84624dc685b401192ca753ff59da016" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.115370 1186 ts_tablet_manager.cc:1918] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:34.115579 1186 log.cc:1198] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1c7a54ce78854137ab2d12e9ce055565
I20250112 20:51:34.122597 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3ade9b410d8443249730cacdf7266eb2 (table default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]) successfully deleted
I20250112 20:51:34.122910 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet bc5af0bc83bc415b972cbbe0fb39f75f (table default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]) successfully deleted
W20250112 20:51:34.123189 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6092a31de9bc47e4bae5f5aebb180c6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092a31de9bc47e4bae5f5aebb180c6d already in progress: creating tablet
I20250112 20:51:34.123301 1858 raft_consensus.cc:491] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:34.123442 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 48e390eb2e404250a8b6edba2ee4f6d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e390eb2e404250a8b6edba2ee4f6d6 already in progress: creating tablet
I20250112 20:51:34.123457 1858 raft_consensus.cc:513] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.124082 1858 leader_election.cc:290] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.125573 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 360) took 1164 ms. Trace:
I20250112 20:51:34.125721 1184 rpcz_store.cc:276] 0112 20:51:32.956957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.960950 (+ 3993us) service_pool.cc:225] Handling call
0112 20:51:34.046464 (+1085514us) ts_tablet_manager.cc:1145] Deleting tablet 31d3669355bc4b6bbc665edaa80c7a9e
0112 20:51:34.046475 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.057635 (+ 11160us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.060763 (+ 3128us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.121175 (+ 60412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085316}
I20250112 20:51:34.127269 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9880e7d303b54d959bca4e8464a6b30c (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.127719 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9880e7d303b54d959bca4e8464a6b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.132661 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0946fe64150e41da957907aa4a6a672f (DEFAULT_TABLE table=default.Table_eadef6eb5b6841f88001ccdad8f9ab3b [id=e203e5c96b52431fa00a10d416d98b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.133086 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946fe64150e41da957907aa4a6a672f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:34.136700 847 leader_election.cc:343] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.136866 847 leader_election.cc:304] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.137158 1858 raft_consensus.cc:2743] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.137308 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_75b794cc68ca4a2c83131bcb146d3b66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.137861 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75b794cc68ca4a2c83131bcb146d3b66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.138747 912 raft_consensus.cc:2463] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:34.145318 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.145736 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.146333 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1e584b761d234c5aa0ab875acd8e1e05 (DEFAULT_TABLE table=default.Table_09bd1e992b194beb825bb3280677c608 [id=27536d1722784720b208bbc108bdc02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.147747 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e584b761d234c5aa0ab875acd8e1e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.148000 1027 tablet_service.cc:1467] Processing CreateTablet for tablet dde64da3dd084a4785986c091852cbc6 (DEFAULT_TABLE table=default.Table_1f79972c8942460f988e2d87f374fc5e [id=97c3a7bf3d874bec9dba134fb054d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.148363 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb (DEFAULT_TABLE table=default.Table_f9f2cdcb247848539893c00bfbef8b18 [id=a40bd70a54ec4854a36809ac58df29b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.148911 1184 tablet_replica.cc:331] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.152874 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f875f075e98e45499291b519caf03184 (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.153432 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f249d70b116443b7823e093efeda5cc3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.153739 1046 raft_consensus.cc:2463] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:34.164656 847 consensus_peers.cc:487] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44f11304909b416aa1e9db1abf633a89. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.166002 847 leader_election.cc:304] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.167114 1858 raft_consensus.cc:2798] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.167243 1858 raft_consensus.cc:491] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.168447 1858 raft_consensus.cc:3054] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.170207 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 (DEFAULT_TABLE table=default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e [id=c0f4f9a776794374b5cfc063becc2876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.170636 1184 raft_consensus.cc:2238] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.171094 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde64da3dd084a4785986c091852cbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.172224 1184 raft_consensus.cc:2267] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.154289 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.177119 1184 ts_tablet_manager.cc:1905] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.177945 1858 raft_consensus.cc:513] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.180991 1012 tablet_service.cc:1467] Processing CreateTablet for tablet b85b9e82f9e847c4acfeecb82201e694 (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.182199 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8fc4a45fe448b919ac001a7980deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.184226 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 104d6b4effb84314a6b2b9db883b4366 (DEFAULT_TABLE table=default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.185828 1020 tablet_service.cc:1467] Processing CreateTablet for tablet be4a237cfeeb4d0fb7ad1abb4ff97845 (DEFAULT_TABLE table=default.Table_0d195bfd940c4601bf584ba32f274f76 [id=5ed2e8826c644775aad86c77003248ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.191228 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (DEFAULT_TABLE table=default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.191551 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f875f075e98e45499291b519caf03184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.191573 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (DEFAULT_TABLE table=default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.192154 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 31d3669355bc4b6bbc665edaa80c7a9e (table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]) successfully deleted
I20250112 20:51:34.192617 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.195371 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f264d84ae4846d1afaaabc41d5789f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.195737 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fd4177f9394a5a82d4f5e02369b3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.197486 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.197814 1824 raft_consensus.cc:491] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.199443 1011 tablet_service.cc:1467] Processing CreateTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 (DEFAULT_TABLE table=default.Table_c8ca9356c6d94a34b8bca0f720df2028 [id=2049ad7ffcee4affa56490ef29e949da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.199890 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0946fe64150e41da957907aa4a6a672f (DEFAULT_TABLE table=default.Table_eadef6eb5b6841f88001ccdad8f9ab3b [id=e203e5c96b52431fa00a10d416d98b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.200417 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 9880e7d303b54d959bca4e8464a6b30c (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.201093 1163 tablet_bootstrap.cc:492] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.201408 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104d6b4effb84314a6b2b9db883b4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.205137 1858 leader_election.cc:290] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.212918 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be4a237cfeeb4d0fb7ad1abb4ff97845. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.213428 1824 raft_consensus.cc:513] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.214418 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f249d70b116443b7823e093efeda5cc3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.215777 1163 ts_tablet_manager.cc:1397] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.214s user 0.014s sys 0.043s
I20250112 20:51:34.215996 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.219702 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9880e7d303b54d959bca4e8464a6b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.220553 1663 leader_election.cc:290] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:34.221208 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153): Couldn't send request to peer 0f5c09e899184de6b770dca873501062. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 20:51:34.219879 1824 leader_election.cc:290] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.222857 1163 raft_consensus.cc:357] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.223145 983 leader_election.cc:343] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.224974 1186 ts_tablet_manager.cc:1939] T 1c7a54ce78854137ab2d12e9ce055565 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.225512 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 331) took 1838 ms. Trace:
I20250112 20:51:34.225615 1186 rpcz_store.cc:276] 0112 20:51:32.387400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.387440 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:33.831367 (+1443927us) ts_tablet_manager.cc:1145] Deleting tablet 1c7a54ce78854137ab2d12e9ce055565
0112 20:51:33.831376 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.014238 (+182862us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.115230 (+100992us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.225467 (+110237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443770}
I20250112 20:51:34.225991 1186 tablet_replica.cc:331] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.226191 1186 raft_consensus.cc:2238] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.226163 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1c7a54ce78854137ab2d12e9ce055565 (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:34.226243 1046 raft_consensus.cc:3054] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.226492 1186 raft_consensus.cc:2267] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.227429 1186 ts_tablet_manager.cc:1905] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.230630 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9880e7d303b54d959bca4e8464a6b30c (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.234671 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.239058 1163 raft_consensus.cc:383] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.240289 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85b9e82f9e847c4acfeecb82201e694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.241261 1638 raft_consensus.cc:491] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.241407 1638 raft_consensus.cc:513] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.242872 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1fd47d1c1a4b42bdbff97f619dc95134
I20250112 20:51:34.243332 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1fd34e829d0645ea80daaff5d27cb266
I20250112 20:51:34.243712 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ec3a27d13db349b4a78c058fdea0ad2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.243964 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_314fc7eb6aff43d7b18dba3be1a7c618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.244091 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_53ad9b3fe99c4e67ad2569718afe2e16" } new_table_name: "default.Table_3300f1667af244cfb434fd02511942b6" modify_external_catalogs: true
W20250112 20:51:34.244277 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec3a27d13db349b4a78c058fdea0ad2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:34.244488 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_314fc7eb6aff43d7b18dba3be1a7c618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.245532 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_28283ef30e624ee992db4e187bd92df0 [id=3a4de1859ffd4db3931ce68febdd8836] alter complete (version 3)
I20250112 20:51:34.247157 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1a635c00d86942f2a0bbd3dfd848b4d4" } modify_external_catalogs: true
I20250112 20:51:34.247334 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1a635c00d86942f2a0bbd3dfd848b4d4" } modify_external_catalogs: true
I20250112 20:51:34.248178 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946fe64150e41da957907aa4a6a672f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.248461 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bea4d83ea644a0930a29f61e7a5fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.248658 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.249008 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.249192 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.249193 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.249449 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.245554 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f76dbd7f681b418ca7bb6c6e8f1f1e58
I20250112 20:51:34.250373 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5dbbb84fc94a4e88bb3331103463e4da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.250921 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dbbb84fc94a4e88bb3331103463e4da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.252866 1163 raft_consensus.cc:738] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.249603 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f0875e271d4aaf9dc5da6aa8113ed0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.254714 1163 consensus_queue.cc:260] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.258741 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.263868 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 49b8cc99742f4f9ba68eb1aa17cab248
I20250112 20:51:34.266680 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.267623 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.271566 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_986d0d39b1fa41abb2166e064df20564" } new_table_name: "default.Table_e55e24b020a64e9ebaf27b249838c000" modify_external_catalogs: true
I20250112 20:51:34.272060 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.273125 1163 ts_tablet_manager.cc:1428] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20250112 20:51:34.274435 1163 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.276130 1163 tablet_bootstrap.cc:654] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.276952 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.279443 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_5cf59aa9a65542c1b2bab56c2238c6a7 [id=75ce02402d924e7f8153b6348a26af9e] alter complete (version 1)
W20250112 20:51:34.284500 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:36661 (request call id 92) took 757 ms (client timeout 1000 ms). Trace:
W20250112 20:51:34.284621 453 rpcz_store.cc:269] 0112 20:51:33.526693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.583163 (+ 56470us) service_pool.cc:225] Handling call
0112 20:51:34.015745 (+432582us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x61b0001f93e8. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 000000000096ceda 00007fa2f13610e8 00007fa2f135c4a1 00007fa2f1356df1 00007fa2f14fb847 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:34.284464 (+268719us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:33.832824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:33.832923 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:33.832928 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:33.832931 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:33.833038 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:33.833052 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:33.833382 (+ 330us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:33.833454 (+ 72us) tablet.cc:698] Row locks acquired
0112 20:51:33.833456 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:33.833489 (+ 33us) write_op.cc:270] Start()
0112 20:51:33.833520 (+ 31us) write_op.cc:276] Timestamp: P: 1736715093833428 usec, L: 0
0112 20:51:33.833524 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:33.833685 (+ 161us) log.cc:844] Serialized 527 byte log entry
0112 20:51:33.843148 (+ 9463us) op_driver.cc:464] REPLICATION: finished
0112 20:51:33.843898 (+ 750us) write_op.cc:301] APPLY: starting
0112 20:51:33.843911 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:33.843946 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:33.843974 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:33.844011 (+ 37us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:33.844043 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:33.844054 (+ 11us) write_op.cc:312] APPLY: finished
0112 20:51:33.844144 (+ 90us) log.cc:844] Serialized 23 byte log entry
0112 20:51:33.844191 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:33.844205 (+ 14us) write_op.cc:454] Released schema lock
0112 20:51:33.844212 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:33.844282 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:33.844288 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 20:51:33.881044 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:33.881158 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:33.881165 (+ 7us) write_op.cc:435] Acquired schema lock
0112 20:51:33.881167 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:33.881277 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:33.881288 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:33.881292 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:33.881395 (+ 103us) tablet.cc:698] Row locks acquired
0112 20:51:33.881399 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:33.881441 (+ 42us) write_op.cc:270] Start()
0112 20:51:33.881479 (+ 38us) write_op.cc:276] Timestamp: P: 1736715093881371 usec, L: 0
0112 20:51:33.881483 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:33.895681 (+ 14198us) log.cc:844] Serialized 356 byte log entry
0112 20:51:33.896231 (+ 550us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 000000000096ceda 00007fa2e986f632 00007fa2eabf7eb5 00007fa2eabf69d9 00007fa2eabfee00 00007fa2eabfec0d 0000000000a71ecc 00007fa2de03cb64 00007fa2de04018b 00007fa2de040008 0000000000a71ecc
0112 20:51:33.937086 (+ 40855us) op_driver.cc:464] REPLICATION: finished
0112 20:51:33.937767 (+ 681us) write_op.cc:301] APPLY: starting
0112 20:51:33.937790 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:33.937843 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:33.937848 (+ 5us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:33.937912 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:33.937954 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:33.937966 (+ 12us) write_op.cc:312] APPLY: finished
0112 20:51:33.938080 (+ 114us) log.cc:844] Serialized 23 byte log entry
0112 20:51:33.938151 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:33.938176 (+ 25us) write_op.cc:454] Released schema lock
0112 20:51:33.938187 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:33.938277 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:33.938283 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"spinlock_wait_cycles":3926272,"child_traces":[["op",{"apply.queue_time_us":713,"apply.run_cpu_time_us":374,"apply.run_wall_time_us":374,"num_ops":1,"prepare.queue_time_us":21859,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":1284,"raft.queue_time_us":179,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":9605}],["op",{"apply.queue_time_us":601,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":495,"num_ops":1,"prepare.queue_time_us":25395,"prepare.run_cpu_time_us":1099,"prepare.run_wall_time_us":16007,"raft.queue_time_us":108,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":204,"replication_time_us":55554,"spinlock_wait_cycles":32291200}]]}
I20250112 20:51:34.285625 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 70c86ad5f59d431e9b1a1fa890fd35c0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.287786 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.289888 1858 raft_consensus.cc:491] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.290187 1858 raft_consensus.cc:513] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.290822 1858 leader_election.cc:290] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.295043 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.295974 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.241142 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0946fe64150e41da957907aa4a6a672f (DEFAULT_TABLE table=default.Table_eadef6eb5b6841f88001ccdad8f9ab3b [id=e203e5c96b52431fa00a10d416d98b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.253509 1663 raft_consensus.cc:491] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.298730 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:34.300098 847 leader_election.cc:343] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.240971 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f249d70b116443b7823e093efeda5cc3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.310964 1663 raft_consensus.cc:513] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.312758 1663 leader_election.cc:290] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:34.313210 1638 leader_election.cc:290] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:34.312990 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9880e7d303b54d959bca4e8464a6b30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.318437 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.319413 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.321352 1186 ts_tablet_manager.cc:1918] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.321489 1186 log.cc:1198] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:34.326365 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.326537 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0946fe64150e41da957907aa4a6a672f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.326735 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_893cdd72c53441f3b51cd4eeb6cda9f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.327356 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:34.330123 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 86695.1us
I20250112 20:51:34.330735 1178 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.330907 1178 ts_tablet_manager.cc:1397] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.244s user 0.056s sys 0.047s
I20250112 20:51:34.331647 1178 raft_consensus.cc:357] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.331862 1178 raft_consensus.cc:383] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.331935 1178 raft_consensus.cc:738] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.332124 1178 consensus_queue.cc:260] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.332988 1178 ts_tablet_manager.cc:1428] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:34.334342 1178 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.336020 1178 tablet_bootstrap.cc:654] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.337215 1858 raft_consensus.cc:491] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.337368 1858 raft_consensus.cc:513] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.337992 1858 leader_election.cc:290] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:34.340600 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36935e6d8c5643388ad163458e2b387c consensus state has no leader
I20250112 20:51:34.343766 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.344587 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.345863 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.346539 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 304e1c8abe094bd99a99fd358b697f81 with delete_type TABLET_DATA_DELETED (replaced by tablet e44422f75a744d078cdf4f0997242e22) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.347807 1046 raft_consensus.cc:2463] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:34.349519 710 leader_election.cc:343] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.350615 847 leader_election.cc:304] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.350966 1858 raft_consensus.cc:2798] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.351109 1858 raft_consensus.cc:695] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.351346 1858 consensus_queue.cc:237] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.352079 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.353710 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.353911 1048 raft_consensus.cc:2463] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.354420 847 leader_election.cc:304] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.354755 1701 raft_consensus.cc:2798] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.354856 1701 raft_consensus.cc:491] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.355378 1701 raft_consensus.cc:3054] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.363519 1164 tablet_bootstrap.cc:654] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.370052 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b3533fddf0f8403bb052fb6b07ab0b20" } new_table_name: "default.Table_42c5efdd34e643cba03f44dc9c276e53" modify_external_catalogs: true
I20250112 20:51:34.372365 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 669e1cfd407e4d87a63e6832d9c6fa1f
I20250112 20:51:34.375348 1824 raft_consensus.cc:491] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.375531 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.375511 1824 raft_consensus.cc:513] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.376032 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.376199 1824 leader_election.cc:290] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.376330 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eadef6eb5b6841f88001ccdad8f9ab3b" } new_table_name: "default.Table_5709a18905f745a48efbc8020384b017" modify_external_catalogs: true
I20250112 20:51:34.378172 710 leader_election.cc:304] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.382341 1745 raft_consensus.cc:2798] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.382364 1184 ts_tablet_manager.cc:1918] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.382490 1184 log.cc:1198] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/304e1c8abe094bd99a99fd358b697f81
I20250112 20:51:34.382524 1745 raft_consensus.cc:695] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:34.382938 1184 ts_tablet_manager.cc:1939] T 304e1c8abe094bd99a99fd358b697f81 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.383682 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 361) took 1426 ms. Trace:
I20250112 20:51:34.383809 1184 rpcz_store.cc:276] 0112 20:51:32.957094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.961180 (+ 4086us) service_pool.cc:225] Handling call
0112 20:51:34.126024 (+1164844us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:34.126035 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.269653 (+143618us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.382247 (+112594us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.383646 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164639}
I20250112 20:51:34.384572 1184 tablet_replica.cc:331] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.383224 1745 consensus_queue.cc:237] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.384867 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f6ac1b3298f0434b9bf1d7ed3e10cb88" } new_table_name: "default.Table_c5a2d4a4380c473bb751bd109b159990" modify_external_catalogs: true
I20250112 20:51:34.385668 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.385900 912 raft_consensus.cc:2463] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:34.386417 983 leader_election.cc:304] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.386713 1824 raft_consensus.cc:2798] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.390076 1824 raft_consensus.cc:491] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.390197 1824 raft_consensus.cc:3054] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.393102 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 304e1c8abe094bd99a99fd358b697f81 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
I20250112 20:51:34.393620 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.394009 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f249d70b116443b7823e093efeda5cc3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.396862 1184 raft_consensus.cc:2238] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:34.398669 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db9178cf782d4bd28eba90cc03796885 consensus state has no leader
I20250112 20:51:34.398949 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f0daaeb0bf6046da8de969b1f4c71c84 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.400413 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 70c86ad5f59d431e9b1a1fa890fd35c0
W20250112 20:51:34.400487 1816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250112 20:51:34.400748 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.400960 453 catalog_manager.cc:6596] ReplaceTablet: tablet 70c86ad5f59d431e9b1a1fa890fd35c0 deleted and replaced by tablet d738720dd7414764b4d1191cdc37895d
I20250112 20:51:34.400960 774 raft_consensus.cc:2463] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.401225 453 master_service.cc:946] ReplaceTablet: received request to replace tablet f3ff8ac6f4d94c10b03a764bbf910c44 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:34.401240 1816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:34.401477 1816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 20:51:34.401507 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.401698 774 raft_consensus.cc:2463] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.405509 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d738720dd7414764b4d1191cdc37895d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.405488 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d5269a5c8c4c049265e18bff4e2c45" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.405699 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d738720dd7414764b4d1191cdc37895d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.405741 912 raft_consensus.cc:2463] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:34.405993 847 leader_election.cc:343] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.406140 847 leader_election.cc:304] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:34.406603 847 leader_election.cc:343] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.406666 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:34.406967 847 leader_election.cc:343] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.407274 1832 raft_consensus.cc:2743] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.407342 847 leader_election.cc:304] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.407639 1832 raft_consensus.cc:2798] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.407807 1832 raft_consensus.cc:491] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.407907 1832 raft_consensus.cc:3054] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.409853 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.410414 710 leader_election.cc:304] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:34.410434 1832 raft_consensus.cc:513] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.411034 1832 leader_election.cc:290] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.411255 454 master_service.cc:946] ReplaceTablet: received request to replace tablet decdec0d34344ec4b422643b0d4e7659 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.411814 1184 raft_consensus.cc:2267] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.413461 1185 ts_tablet_manager.cc:1918] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:34.413692 1185 log.cc:1198] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7bd84ab3a8fc4580b971d6b9adb18d6f
I20250112 20:51:34.414290 1185 ts_tablet_manager.cc:1939] T 7bd84ab3a8fc4580b971d6b9adb18d6f P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.414347 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.417122 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.417325 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.418954 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d5269a5c8c4c049265e18bff4e2c45" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:34.419404 983 leader_election.cc:343] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.419526 983 leader_election.cc:304] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.419745 455 master_service.cc:946] ReplaceTablet: received request to replace tablet e72fda1743fc4f96b29add6e36e884de from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:34.420440 983 leader_election.cc:343] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.420680 1821 raft_consensus.cc:2743] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.422547 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7f164d4ff332493e885b6ef5272e508a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.422955 1824 raft_consensus.cc:513] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.423121 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f164d4ff332493e885b6ef5272e508a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.424294 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:34.424503 912 raft_consensus.cc:3054] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.423017 1932 raft_consensus.cc:2798] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.426797 1932 raft_consensus.cc:491] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.426915 1932 raft_consensus.cc:3054] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.427335 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:34.427867 847 leader_election.cc:343] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.429611 1932 raft_consensus.cc:513] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.430217 1932 leader_election.cc:290] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:34.430697 1047 raft_consensus.cc:2463] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:34.430912 1816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:34.431115 1816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250112 20:51:34.431034 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4e304a1212412f8e99bfd222a110b5" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:34.431253 1816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:34.431377 1816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:34.431468 1824 leader_election.cc:290] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.431886 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 310) took 2217 ms. Trace:
I20250112 20:51:34.431980 1185 rpcz_store.cc:276] 0112 20:51:32.214397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.214472 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:34.036684 (+1822212us) ts_tablet_manager.cc:1145] Deleting tablet 7bd84ab3a8fc4580b971d6b9adb18d6f
0112 20:51:34.036693 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.376043 (+339350us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.413322 (+ 37279us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.431836 (+ 18514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1811546}
I20250112 20:51:34.432407 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7bd84ab3a8fc4580b971d6b9adb18d6f (table default.Table_96bb34aa833040378700d550b97d143e [id=8f418af80a3b4cd78aea9984451558ed]) successfully deleted
W20250112 20:51:34.423247 710 leader_election.cc:343] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:34.432978 710 consensus_peers.cc:487] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f769533bfde0448792d01b883e41763c. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:34.433177 710 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:34.433841 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d5269a5c8c4c049265e18bff4e2c45" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:34.434070 911 raft_consensus.cc:3054] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:34.434402 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2664a2e98f74f73ab2dc0adaa2335d3 consensus state has no leader
W20250112 20:51:34.435005 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0946fe64150e41da957907aa4a6a672f consensus state has no leader
I20250112 20:51:34.435137 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d738720dd7414764b4d1191cdc37895d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.434855 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7f122343e2d74bf7b3eefea9de3ff725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.435586 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f122343e2d74bf7b3eefea9de3ff725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.435740 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.435945 1045 raft_consensus.cc:2463] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:34.436358 710 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.436568 710 leader_election.cc:304] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:34.436722 1184 ts_tablet_manager.cc:1905] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.436854 1932 raft_consensus.cc:2743] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.437430 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.437628 774 raft_consensus.cc:3054] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.440539 774 raft_consensus.cc:2463] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:34.441052 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 235faeb3c35d46d2882ffcd8cb63ddb8 consensus state has no leader
I20250112 20:51:34.441424 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.446689 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.447126 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 325) took 2062 ms. Trace:
I20250112 20:51:34.447268 1185 rpcz_store.cc:276] 0112 20:51:32.369859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.369904 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:34.432644 (+2062740us) ts_tablet_manager.cc:1145] Deleting tablet 1a04c223889f424e831e82a52a84f804
0112 20:51:34.432653 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.432752 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062549}
W20250112 20:51:34.447779 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1a04c223889f424e831e82a52a84f804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a04c223889f424e831e82a52a84f804 already in progress: creating tablet
I20250112 20:51:34.448236 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 326) took 2077 ms. Trace:
I20250112 20:51:34.448383 1185 rpcz_store.cc:276] 0112 20:51:32.370656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.370703 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:34.448103 (+2077400us) ts_tablet_manager.cc:1145] Deleting tablet d0018e1eabf24241a358a7e00b6b5907
0112 20:51:34.448113 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.448191 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2077215}
W20250112 20:51:34.448845 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d0018e1eabf24241a358a7e00b6b5907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0018e1eabf24241a358a7e00b6b5907 already in progress: creating tablet
I20250112 20:51:34.449127 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f0daaeb0bf6046da8de969b1f4c71c84
I20250112 20:51:34.449538 451 catalog_manager.cc:6596] ReplaceTablet: tablet f0daaeb0bf6046da8de969b1f4c71c84 deleted and replaced by tablet ab9be1f065ec4605a0a3fe69770793d0
I20250112 20:51:34.450330 847 leader_election.cc:304] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.450634 1704 raft_consensus.cc:2798] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.450789 1704 raft_consensus.cc:695] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.450855 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:34.451031 847 consensus_peers.cc:487] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c38c8594a384689ae57c26174765284. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.451100 1704 consensus_queue.cc:237] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.451447 983 leader_election.cc:343] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.451795 710 leader_election.cc:304] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:34.452227 1932 raft_consensus.cc:2798] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.452363 1932 raft_consensus.cc:491] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.452533 1932 raft_consensus.cc:3054] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.454713 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f594de5fea864b98a7b2838bf0257f3c" } modify_external_catalogs: true
I20250112 20:51:34.454771 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.454914 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f594de5fea864b98a7b2838bf0257f3c" } modify_external_catalogs: true
I20250112 20:51:34.455504 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a1b5b3029a0f4295904392987db19062"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.455986 1186 ts_tablet_manager.cc:1939] T bc5af0bc83bc415b972cbbe0fb39f75f P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:34.456073 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1b5b3029a0f4295904392987db19062 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.456151 1932 raft_consensus.cc:513] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.456872 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 333) took 2068 ms. Trace:
I20250112 20:51:34.456991 1932 leader_election.cc:290] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:34.457780 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8284ddd01ebb44d49bfeac291415f6b1" } modify_external_catalogs: true
I20250112 20:51:34.457952 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8284ddd01ebb44d49bfeac291415f6b1" } modify_external_catalogs: true
I20250112 20:51:34.459098 1701 raft_consensus.cc:513] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.460757 1701 leader_election.cc:290] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.462934 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 328) took 2061 ms. Trace:
I20250112 20:51:34.463203 1185 rpcz_store.cc:276] 0112 20:51:32.401324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.401373 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:34.462762 (+2061389us) ts_tablet_manager.cc:1145] Deleting tablet 83c13126e53b46c0a1e7beafe1b0d295
0112 20:51:34.462774 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.462881 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061119}
I20250112 20:51:34.463626 1185 tablet_replica.cc:331] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:34.463757 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 83c13126e53b46c0a1e7beafe1b0d295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c13126e53b46c0a1e7beafe1b0d295 already in progress: creating tablet
I20250112 20:51:34.463959 1185 raft_consensus.cc:2238] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.464239 1185 raft_consensus.cc:2267] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.464851 1185 ts_tablet_manager.cc:1905] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.427083 912 raft_consensus.cc:2463] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:34.466017 1816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=44167
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:34.467403 911 raft_consensus.cc:2463] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:34.467610 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.468060 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d5269a5c8c4c049265e18bff4e2c45" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.468254 1045 raft_consensus.cc:3054] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.468828 1816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:34.469425 983 leader_election.cc:304] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.470342 1048 raft_consensus.cc:3054] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.470767 1163 tablet_bootstrap.cc:492] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.470963 1816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:51:34.471269 1163 ts_tablet_manager.cc:1397] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.198s user 0.005s sys 0.010s
I20250112 20:51:34.471083 1824 raft_consensus.cc:2798] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.471803 1824 raft_consensus.cc:695] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:34.472183 1824 consensus_queue.cc:237] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.473364 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:34.473949 710 leader_election.cc:304] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:34.474000 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.474234 1047 raft_consensus.cc:3054] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.470922 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.478160 1701 consensus_queue.cc:1035] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.480082 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:34.480571 453 catalog_manager.cc:6596] ReplaceTablet: tablet f3ff8ac6f4d94c10b03a764bbf910c44 deleted and replaced by tablet 691e2c3f72ab4f439678d166b9b31661
I20250112 20:51:34.480981 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ac72a7aedf80477ea4d02c7a2f150f28" } modify_external_catalogs: true
I20250112 20:51:34.481216 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ac72a7aedf80477ea4d02c7a2f150f28" } modify_external_catalogs: true
I20250112 20:51:34.481806 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet decdec0d34344ec4b422643b0d4e7659
I20250112 20:51:34.482275 454 catalog_manager.cc:6596] ReplaceTablet: tablet decdec0d34344ec4b422643b0d4e7659 deleted and replaced by tablet db9bab2477a94c169817d9de8aaad1c0
I20250112 20:51:34.482612 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4c02bae8e4d64469b992b13dd36c7e81" } modify_external_catalogs: true
I20250112 20:51:34.482846 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4c02bae8e4d64469b992b13dd36c7e81" } modify_external_catalogs: true
I20250112 20:51:34.489452 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.489832 874 tablet_service.cc:1514] Processing DeleteTablet for tablet decdec0d34344ec4b422643b0d4e7659 with delete_type TABLET_DATA_DELETED (replaced by tablet db9bab2477a94c169817d9de8aaad1c0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.489873 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet decdec0d34344ec4b422643b0d4e7659 with delete_type TABLET_DATA_DELETED (replaced by tablet db9bab2477a94c169817d9de8aaad1c0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.490082 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.492105 1163 raft_consensus.cc:357] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.492372 1163 raft_consensus.cc:383] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.492491 1163 raft_consensus.cc:738] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.493028 1163 consensus_queue.cc:260] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.497423 1045 raft_consensus.cc:2463] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:34.497679 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.497885 745 tablet_service.cc:1514] Processing DeleteTablet for tablet decdec0d34344ec4b422643b0d4e7659 with delete_type TABLET_DATA_DELETED (replaced by tablet db9bab2477a94c169817d9de8aaad1c0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.497524 1932 raft_consensus.cc:2798] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.498392 1932 raft_consensus.cc:695] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:34.498770 1932 consensus_queue.cc:237] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.499215 710 leader_election.cc:343] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.500102 1048 raft_consensus.cc:2463] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.500612 847 leader_election.cc:304] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.501227 1701 raft_consensus.cc:2798] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.501384 1701 raft_consensus.cc:695] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.501636 1701 consensus_queue.cc:237] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.501791 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c14102aecb743e2a82660039bd0f038" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.501991 774 raft_consensus.cc:3054] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.504312 1047 raft_consensus.cc:2463] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:34.504891 1821 raft_consensus.cc:491] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.505039 1821 raft_consensus.cc:513] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.505331 1178 tablet_bootstrap.cc:492] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.505489 1178 ts_tablet_manager.cc:1397] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.171s user 0.005s sys 0.006s
I20250112 20:51:34.505661 1821 leader_election.cc:290] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.505868 774 raft_consensus.cc:2463] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.506635 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.506954 911 raft_consensus.cc:2388] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:34.507422 1178 raft_consensus.cc:357] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.507643 1178 raft_consensus.cc:383] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.507738 1178 raft_consensus.cc:738] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.507969 1178 consensus_queue.cc:260] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.508529 1178 ts_tablet_manager.cc:1428] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:34.508865 1178 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.510752 1178 tablet_bootstrap.cc:654] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.512684 1704 consensus_queue.cc:1035] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.512735 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_65c2866d97af4a62ab44bf88f718b319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.513139 1704 consensus_queue.cc:1035] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:34.513433 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c2866d97af4a62ab44bf88f718b319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.513633 710 leader_election.cc:304] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:34.514014 1931 raft_consensus.cc:2798] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.514164 1931 raft_consensus.cc:695] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:34.514482 1931 consensus_queue.cc:237] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.516952 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1aac31fd32400b9094f8e2de4fe42c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.518366 1163 ts_tablet_manager.cc:1428] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250112 20:51:34.518784 1821 raft_consensus.cc:491] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.518957 1821 raft_consensus.cc:513] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.519603 1821 leader_election.cc:290] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.520607 1163 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.520531 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:34.521109 983 leader_election.cc:343] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.521376 983 leader_election.cc:304] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.521853 1821 raft_consensus.cc:3054] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.525019 1821 raft_consensus.cc:2743] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.526444 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1e49d5313296492aa6576d85862799b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.526979 1184 ts_tablet_manager.cc:1918] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:34.527004 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e49d5313296492aa6576d85862799b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.527122 1184 log.cc:1198] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1c7a54ce78854137ab2d12e9ce055565
I20250112 20:51:34.527472 1184 ts_tablet_manager.cc:1939] T 1c7a54ce78854137ab2d12e9ce055565 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.528276 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 362) took 1571 ms. Trace:
I20250112 20:51:34.528378 1184 rpcz_store.cc:276] 0112 20:51:32.957212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.961415 (+ 4203us) service_pool.cc:225] Handling call
0112 20:51:34.384503 (+1423088us) ts_tablet_manager.cc:1145] Deleting tablet 1c7a54ce78854137ab2d12e9ce055565
0112 20:51:34.384514 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.443820 (+ 59306us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.526863 (+ 83043us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.528226 (+ 1363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1422931}
I20250112 20:51:34.528847 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1c7a54ce78854137ab2d12e9ce055565 (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:34.456980 1186 rpcz_store.cc:276] 0112 20:51:32.388053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.388526 (+ 473us) service_pool.cc:225] Handling call
0112 20:51:34.225910 (+1837384us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:34.225919 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.318326 (+ 92407us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.321231 (+ 2905us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.456832 (+135601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1837185}
I20250112 20:51:34.530156 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet bc5af0bc83bc415b972cbbe0fb39f75f (table default.Table_cc561e5052ca4268b3733b33a981bf86 [id=cc2a0566421c4f659bbe83d168cfbb4f]) successfully deleted
I20250112 20:51:34.530449 1186 tablet_replica.cc:331] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.530666 1186 raft_consensus.cc:2238] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.530843 1186 raft_consensus.cc:2267] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.531700 1186 ts_tablet_manager.cc:1905] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.533000 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:34.534981 983 leader_election.cc:343] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.535187 983 leader_election.cc:304] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.535759 1821 raft_consensus.cc:2743] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.547590 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_77fce9ea810348bea460274d6f69db14" } new_table_name: "default.Table_15e3fd8e71984a81acb2dabf8f3563f7" modify_external_catalogs: true
W20250112 20:51:34.553555 983 consensus_peers.cc:487] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d770e0399e80491f8de9b08d1eb2f139. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.555521 1164 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:34.555711 1164 ts_tablet_manager.cc:1397] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.472s user 0.008s sys 0.011s
I20250112 20:51:34.556895 1164 raft_consensus.cc:357] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.557186 1164 raft_consensus.cc:383] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.557262 1164 raft_consensus.cc:738] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.559350 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 9570269368e14b1abeaffdcc741c6f3a (DEFAULT_TABLE table=default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.559804 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9570269368e14b1abeaffdcc741c6f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.560304 1185 ts_tablet_manager.cc:1918] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:34.561058 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 480dca3c3a1e4da7a84834e02f26b299 (DEFAULT_TABLE table=default.Table_626c18b182014d509b9de95803204412 [id=6a89b49f4d5c462a8e539c1801293c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.561445 1185 log.cc:1198] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:34.561729 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480dca3c3a1e4da7a84834e02f26b299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.561868 877 tablet_service.cc:1467] Processing CreateTablet for tablet 9570269368e14b1abeaffdcc741c6f3a (DEFAULT_TABLE table=default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.562254 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9570269368e14b1abeaffdcc741c6f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.562366 1185 ts_tablet_manager.cc:1939] T ac4e304a1212412f8e99bfd222a110b5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.565048 874 tablet_service.cc:1467] Processing CreateTablet for tablet 480dca3c3a1e4da7a84834e02f26b299 (DEFAULT_TABLE table=default.Table_626c18b182014d509b9de95803204412 [id=6a89b49f4d5c462a8e539c1801293c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.565428 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480dca3c3a1e4da7a84834e02f26b299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.569145 884 tablet_service.cc:1467] Processing CreateTablet for tablet d80791d8582446b89adf212d3d6507eb (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.569561 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80791d8582446b89adf212d3d6507eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.574071 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d80791d8582446b89adf212d3d6507eb (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.574489 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80791d8582446b89adf212d3d6507eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.574584 885 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (DEFAULT_TABLE table=default.Table_80d55cc666794ca8853848350b2cb0e6 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.574918 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6b2c4ea5f4167896c78b1c64cbb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.577353 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (DEFAULT_TABLE table=default.Table_80d55cc666794ca8853848350b2cb0e6 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.577719 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6b2c4ea5f4167896c78b1c64cbb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.573523 1164 consensus_queue.cc:260] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.581779 1164 ts_tablet_manager.cc:1428] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250112 20:51:34.581894 882 tablet_service.cc:1467] Processing CreateTablet for tablet fb09123a64e744c89cb8343fc27d8d60 (DEFAULT_TABLE table=default.Table_4246de22bb514400b9eb57117e36a6f9 [id=55b0f31b541646faa7a63e7f80a15dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.581928 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 549d92191a594788bf639cd60a6b5142 (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.582229 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549d92191a594788bf639cd60a6b5142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.582247 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb09123a64e744c89cb8343fc27d8d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.582281 1164 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.585050 886 tablet_service.cc:1467] Processing CreateTablet for tablet 549d92191a594788bf639cd60a6b5142 (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.585376 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549d92191a594788bf639cd60a6b5142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.585903 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 329) took 2184 ms. Trace:
I20250112 20:51:34.586058 1185 rpcz_store.cc:276] 0112 20:51:32.401429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.401492 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:34.463540 (+2062048us) ts_tablet_manager.cc:1145] Deleting tablet ac4e304a1212412f8e99bfd222a110b5
0112 20:51:34.463549 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.471010 (+ 7461us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.560144 (+ 89134us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.585860 (+ 25716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061827}
I20250112 20:51:34.586519 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 327) took 2185 ms. Trace:
I20250112 20:51:34.586622 1185 rpcz_store.cc:276] 0112 20:51:32.401219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.405202 (+ 3983us) service_pool.cc:225] Handling call
0112 20:51:34.586360 (+2181158us) ts_tablet_manager.cc:1145] Deleting tablet 81b932b7e1e34d4ab18aae40047dc038
0112 20:51:34.586372 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.586487 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2180962}
I20250112 20:51:34.586962 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 340) took 2167 ms. Trace:
I20250112 20:51:34.588966 1012 tablet_service.cc:1467] Processing CreateTablet for tablet fb09123a64e744c89cb8343fc27d8d60 (DEFAULT_TABLE table=default.Table_4246de22bb514400b9eb57117e36a6f9 [id=55b0f31b541646faa7a63e7f80a15dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.588943 1185 rpcz_store.cc:276] 0112 20:51:32.419841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.419888 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:34.586859 (+2166971us) ts_tablet_manager.cc:1145] Deleting tablet ad1a4b5ac2f94874b99b3ef790a034fd
0112 20:51:34.586869 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.586938 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130964}
I20250112 20:51:34.589257 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7038f1cfb0f944378fdf6ab8daa15d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.589792 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb09123a64e744c89cb8343fc27d8d60. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:34.589785 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7038f1cfb0f944378fdf6ab8daa15d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.590083 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 339) took 2170 ms. Trace:
I20250112 20:51:34.590205 1185 rpcz_store.cc:276] 0112 20:51:32.419358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.419652 (+ 294us) service_pool.cc:225] Handling call
0112 20:51:34.589256 (+2169604us) ts_tablet_manager.cc:1145] Deleting tablet 8d08048c402345a686d1e3fb3039bbc1
0112 20:51:34.589267 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.589367 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2128830}
I20250112 20:51:34.590552 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 359) took 2007 ms. Trace:
I20250112 20:51:34.590656 1185 rpcz_store.cc:276] 0112 20:51:32.583486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.678984 (+ 95498us) service_pool.cc:225] Handling call
0112 20:51:34.590444 (+1911460us) ts_tablet_manager.cc:1145] Deleting tablet d770e0399e80491f8de9b08d1eb2f139
0112 20:51:34.590455 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.590523 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911245}
I20250112 20:51:34.593950 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 360) took 1995 ms. Trace:
I20250112 20:51:34.594424 1185 rpcz_store.cc:276] 0112 20:51:32.598461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.679219 (+ 80758us) service_pool.cc:225] Handling call
0112 20:51:34.593836 (+1914617us) ts_tablet_manager.cc:1145] Deleting tablet 4c3f359655c94f1d905943069548845b
0112 20:51:34.593848 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.593917 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1914462}
I20250112 20:51:34.595000 1185 tablet_replica.cc:331] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:34.595443 1185 raft_consensus.cc:2238] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.595903 1185 raft_consensus.cc:2267] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.593086 1164 tablet_bootstrap.cc:654] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.600203 1185 ts_tablet_manager.cc:1905] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.602667 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ac4e304a1212412f8e99bfd222a110b5 (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:34.603644 745 tablet_service.cc:1467] Processing CreateTablet for tablet 9570269368e14b1abeaffdcc741c6f3a (DEFAULT_TABLE table=default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.603729 741 tablet_service.cc:1467] Processing CreateTablet for tablet 480dca3c3a1e4da7a84834e02f26b299 (DEFAULT_TABLE table=default.Table_626c18b182014d509b9de95803204412 [id=6a89b49f4d5c462a8e539c1801293c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.604056 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9570269368e14b1abeaffdcc741c6f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.604084 754 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (DEFAULT_TABLE table=default.Table_80d55cc666794ca8853848350b2cb0e6 [id=992620c02c1b401a8936586a1ef0d291]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:34.604732 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ad1a4b5ac2f94874b99b3ef790a034fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad1a4b5ac2f94874b99b3ef790a034fd already in progress: creating tablet
W20250112 20:51:34.605044 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8d08048c402345a686d1e3fb3039bbc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d08048c402345a686d1e3fb3039bbc1 already in progress: creating tablet
W20250112 20:51:34.605403 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4c3f359655c94f1d905943069548845b because the tablet was not found. No further retry: Not found: Tablet not found: 4c3f359655c94f1d905943069548845b
I20250112 20:51:34.607596 752 tablet_service.cc:1467] Processing CreateTablet for tablet d80791d8582446b89adf212d3d6507eb (DEFAULT_TABLE table=default.Table_30fc56d0c044444a9a07861b6650fac6 [id=abd0f618fc7b449db908c4865a4538d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.608139 753 tablet_service.cc:1467] Processing CreateTablet for tablet fb09123a64e744c89cb8343fc27d8d60 (DEFAULT_TABLE table=default.Table_4246de22bb514400b9eb57117e36a6f9 [id=55b0f31b541646faa7a63e7f80a15dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.608641 737 tablet_service.cc:1467] Processing CreateTablet for tablet 549d92191a594788bf639cd60a6b5142 (DEFAULT_TABLE table=default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:34.609889 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6b2c4ea5f4167896c78b1c64cbb62. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:34.612270 1956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:34.613297 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d80791d8582446b89adf212d3d6507eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.615607 1163 tablet_bootstrap.cc:654] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.617110 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb09123a64e744c89cb8343fc27d8d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.623159 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 549d92191a594788bf639cd60a6b5142. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:34.619848 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:34.625092 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480dca3c3a1e4da7a84834e02f26b299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:34.625526 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 367) took 1667 ms. Trace:
I20250112 20:51:34.625635 1184 rpcz_store.cc:276] 0112 20:51:32.957784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.961602 (+ 3818us) service_pool.cc:225] Handling call
0112 20:51:34.625369 (+1663767us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:34.625378 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.625472 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663575}
W20250112 20:51:34.626050 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3343e1d7f1d04a498bd0ee00311b6a61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3343e1d7f1d04a498bd0ee00311b6a61 already in progress: creating tablet
I20250112 20:51:34.627332 1184 tablet_replica.cc:331] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.628021 1184 raft_consensus.cc:2238] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.628263 1184 raft_consensus.cc:2267] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.628930 1184 ts_tablet_manager.cc:1905] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.637275 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_326496344cf44e9d949ee3545f916c40" } modify_external_catalogs: true
I20250112 20:51:34.637466 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_326496344cf44e9d949ee3545f916c40" } modify_external_catalogs: true
W20250112 20:51:34.640897 847 consensus_peers.cc:487] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.641664 1164 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:34.641806 1164 ts_tablet_manager.cc:1397] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.004s
I20250112 20:51:34.642625 1178 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.642621 1164 raft_consensus.cc:357] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.642810 1164 raft_consensus.cc:383] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.644431 1816 server_base.cc:1034] running on GCE node
I20250112 20:51:34.647796 1163 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.648057 1163 ts_tablet_manager.cc:1397] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.127s user 0.003s sys 0.000s
W20250112 20:51:34.649036 1958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:34.648959 1163 raft_consensus.cc:357] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.649174 1163 raft_consensus.cc:383] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.649293 1163 raft_consensus.cc:738] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.649610 1163 consensus_queue.cc:260] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.650213 1178 ts_tablet_manager.cc:1397] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.000s
I20250112 20:51:34.650362 1816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:34.651185 1178 raft_consensus.cc:357] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.651377 1178 raft_consensus.cc:383] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.651461 1178 raft_consensus.cc:738] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.651705 1178 consensus_queue.cc:260] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.652189 1178 ts_tablet_manager.cc:1428] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:34.652817 1178 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:34.646288 710 consensus_peers.cc:487] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.657532 1164 raft_consensus.cc:738] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.657845 1164 consensus_queue.cc:260] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.660094 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9b86e02c3663476d9f4b850bf5982df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.660405 1164 ts_tablet_manager.cc:1428] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
W20250112 20:51:34.660650 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b86e02c3663476d9f4b850bf5982df8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.660668 1164 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.650238 1163 ts_tablet_manager.cc:1428] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:34.661610 1163 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.661877 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 61b74aaf7f5b4c87acbefaa4df5c10d2
I20250112 20:51:34.662658 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_455466a0838742f3a2cc35cdab729433"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.663115 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_455466a0838742f3a2cc35cdab729433 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.663172 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.667321 1164 tablet_bootstrap.cc:654] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.668028 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.669193 1185 ts_tablet_manager.cc:1918] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.669373 1185 log.cc:1198] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/00c0712ab2de43eea8c77ab39a971a2c
I20250112 20:51:34.669817 1185 ts_tablet_manager.cc:1939] T 00c0712ab2de43eea8c77ab39a971a2c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.670053 1163 tablet_bootstrap.cc:654] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.671038 1816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:34.671630 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 361) took 2073 ms. Trace:
I20250112 20:51:34.674288 1185 rpcz_store.cc:276] 0112 20:51:32.665611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.679381 (+ 13770us) service_pool.cc:225] Handling call
0112 20:51:34.594936 (+1915555us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:34.594945 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.654670 (+ 59725us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.669085 (+ 14415us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.671577 (+ 2492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1915376}
I20250112 20:51:34.680080 1186 ts_tablet_manager.cc:1918] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.680238 1186 log.cc:1198] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3ade9b410d8443249730cacdf7266eb2
I20250112 20:51:34.680575 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 00c0712ab2de43eea8c77ab39a971a2c (table default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]) successfully deleted
I20250112 20:51:34.681172 1185 tablet_replica.cc:331] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:34.682226 1185 raft_consensus.cc:2238] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.682651 1185 raft_consensus.cc:2267] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.683430 1185 ts_tablet_manager.cc:1905] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.685490 1186 ts_tablet_manager.cc:1939] T 3ade9b410d8443249730cacdf7266eb2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.685883 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 334) took 2296 ms. Trace:
I20250112 20:51:34.685973 1186 rpcz_store.cc:276] 0112 20:51:32.388877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.388947 (+ 70us) service_pool.cc:225] Handling call
0112 20:51:34.530379 (+2141432us) ts_tablet_manager.cc:1145] Deleting tablet 3ade9b410d8443249730cacdf7266eb2
0112 20:51:34.530390 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.647159 (+116769us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.679950 (+ 32791us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.685797 (+ 5847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2141231}
I20250112 20:51:34.686269 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.686455 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3ade9b410d8443249730cacdf7266eb2 (table default.Table_65e847e586ae4fc99dee908b16e66be2 [id=7b33423047374e888a01f3c9b3861ed3]) successfully deleted
I20250112 20:51:34.686762 1816 hybrid_clock.cc:648] HybridClock initialized: now 1736715094686582 us; error 137 us; skew 500 ppm
I20250112 20:51:34.687209 1816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:34.690279 1164 tablet_bootstrap.cc:492] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:34.690768 1164 ts_tablet_manager.cc:1397] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20250112 20:51:34.691574 1164 raft_consensus.cc:357] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.691917 1164 raft_consensus.cc:383] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.692028 1164 raft_consensus.cc:738] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.692325 1164 consensus_queue.cc:260] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.696316 1824 consensus_queue.cc:1035] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.701385 1178 tablet_bootstrap.cc:654] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.705832 1164 ts_tablet_manager.cc:1428] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250112 20:51:34.709041 1164 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.711328 1184 ts_tablet_manager.cc:1918] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.712057 1184 log.cc:1198] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/158cfe5ba52c4771af97841885b29f9d
I20250112 20:51:34.712656 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 70653110aec74727acec9d63c2eb0f05
I20250112 20:51:34.713802 1164 tablet_bootstrap.cc:654] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.713821 1701 raft_consensus.cc:491] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.713970 1701 raft_consensus.cc:513] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.714700 1701 leader_election.cc:290] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:34.715488 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.716574 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.716938 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:34.717408 847 leader_election.cc:343] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.717398 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.717586 774 raft_consensus.cc:2463] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.718184 847 leader_election.cc:304] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.719050 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.719249 1701 raft_consensus.cc:2798] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.719245 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fd99ee08b68344f499dba32ccf27ff4a" } new_table_name: "default.Table_be8270d1ca354e748dac253a4aeb09a3" modify_external_catalogs: true
I20250112 20:51:34.719352 1701 raft_consensus.cc:491] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.719439 1701 raft_consensus.cc:3054] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.719956 1858 raft_consensus.cc:491] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.720134 1858 raft_consensus.cc:513] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.720883 1858 leader_election.cc:290] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.721491 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.721844 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480d49569886408a8ce781a93bb5d10d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.722075 1046 raft_consensus.cc:2388] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:34.722517 847 leader_election.cc:304] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:34.724236 983 consensus_peers.cc:487] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.724694 1701 raft_consensus.cc:513] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.730036 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a3ca4d49496e42ddb263264ceef2f27a
I20250112 20:51:34.731603 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.732427 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.734544 1701 leader_election.cc:290] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:34.736790 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_66ed5acb10ac4640a299d0b36ebc314b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.737388 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66ed5acb10ac4640a299d0b36ebc314b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.737653 1832 raft_consensus.cc:491] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.737793 1832 raft_consensus.cc:513] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.738391 1832 leader_election.cc:290] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:34.738487 1185 ts_tablet_manager.cc:1918] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:34.738538 1832 raft_consensus.cc:3054] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.738603 1185 log.cc:1198] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3a2ee62fff7d4cf7b4aaeacf63ed8e02
I20250112 20:51:34.738976 1185 ts_tablet_manager.cc:1939] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.739413 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.739724 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 363) took 2073 ms. Trace:
I20250112 20:51:34.739833 1185 rpcz_store.cc:276] 0112 20:51:32.665867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.679899 (+ 14032us) service_pool.cc:225] Handling call
0112 20:51:34.681064 (+2001165us) ts_tablet_manager.cc:1145] Deleting tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02
0112 20:51:34.681079 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.732234 (+ 51155us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.738354 (+ 6120us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.739693 (+ 1339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2000991}
I20250112 20:51:34.740480 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7293510f40804e2aabccf92986489dab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.740667 775 raft_consensus.cc:2463] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.741428 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.741509 847 leader_election.cc:304] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:34.741791 1858 raft_consensus.cc:2798] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.741894 1858 raft_consensus.cc:491] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.741972 1858 raft_consensus.cc:3054] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.742906 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7293510f40804e2aabccf92986489dab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.743147 1048 raft_consensus.cc:2463] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.744278 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.744453 1858 raft_consensus.cc:513] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.744850 774 raft_consensus.cc:3054] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.745157 1858 leader_election.cc:290] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:34.746137 847 leader_election.cc:343] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.746379 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7293510f40804e2aabccf92986489dab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.746621 1048 raft_consensus.cc:3054] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.746798 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7293510f40804e2aabccf92986489dab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.746971 775 raft_consensus.cc:3054] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.748818 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:34.749641 775 raft_consensus.cc:2463] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.749786 1048 raft_consensus.cc:2463] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.750154 847 leader_election.cc:304] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:34.750479 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.750720 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.751169 1832 raft_consensus.cc:2743] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.751281 1858 raft_consensus.cc:2798] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.752523 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:34.753726 847 consensus_peers.cc:487] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c38c8594a384689ae57c26174765284. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.753798 1858 raft_consensus.cc:695] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.754153 1858 consensus_queue.cc:237] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.755880 774 raft_consensus.cc:2463] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.756502 847 leader_election.cc:304] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.756630 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_23882b539e614fcca89f4d9b2fadd948"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:34.756948 1164 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:34.757086 1164 ts_tablet_manager.cc:1397] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
W20250112 20:51:34.757262 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23882b539e614fcca89f4d9b2fadd948 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.757772 1164 raft_consensus.cc:357] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.757961 1164 raft_consensus.cc:383] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.758047 1164 raft_consensus.cc:738] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.759601 1701 raft_consensus.cc:2798] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.759833 1701 raft_consensus.cc:695] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.760169 1701 consensus_queue.cc:237] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.770625 1186 tablet_replica.cc:331] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.770859 1186 raft_consensus.cc:2238] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.771229 1186 raft_consensus.cc:2267] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.771837 1186 ts_tablet_manager.cc:1905] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.773927 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65c2866d97af4a62ab44bf88f718b319" } new_table_name: "default.Table_6e73aeaec7af4cd0952e8f427103f61f" modify_external_catalogs: true
I20250112 20:51:34.779732 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_6476b4820a6f465f9b721e9bd7b463d2 [id=c6a4acd610364581857563f69a97d6a3] alter complete (version 1)
I20250112 20:51:34.781374 752 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.781672 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4e304a1212412f8e99bfd222a110b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.781790 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.781677 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0712ab2de43eea8c77ab39a971a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:34.786773 1184 ts_tablet_manager.cc:1939] T 158cfe5ba52c4771af97841885b29f9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.787842 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.788067 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.787853 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.787652 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 366) took 1829 ms. Trace:
I20250112 20:51:34.788343 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:34.788399 1184 rpcz_store.cc:276] 0112 20:51:32.957671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.960706 (+ 3035us) service_pool.cc:225] Handling call
0112 20:51:34.627258 (+1666552us) ts_tablet_manager.cc:1145] Deleting tablet 158cfe5ba52c4771af97841885b29f9d
0112 20:51:34.627265 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.631941 (+ 4676us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.711183 (+ 79242us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.787606 (+ 76423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663428}
I20250112 20:51:34.788946 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 386) took 1577 ms. Trace:
I20250112 20:51:34.789076 1184 rpcz_store.cc:276] 0112 20:51:33.211862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.218923 (+ 7061us) service_pool.cc:225] Handling call
0112 20:51:34.788789 (+1569866us) ts_tablet_manager.cc:1145] Deleting tablet e72fda1743fc4f96b29add6e36e884de
0112 20:51:34.788798 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.788907 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547342}
I20250112 20:51:34.789765 1184 tablet_replica.cc:331] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.790028 1184 raft_consensus.cc:2238] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.790203 1184 raft_consensus.cc:2267] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.790407 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_80d55cc666794ca8853848350b2cb0e6" } new_table_name: "default.Table_cc55237c53344dfbbf00a7b33b6962e9" modify_external_catalogs: true
I20250112 20:51:34.790409 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 158cfe5ba52c4771af97841885b29f9d (table default.Table_9d07ceff6ea140e997aff9afbaf71e91 [id=eb21d2c143f149cbb2702e1a43105d95]) successfully deleted
W20250112 20:51:34.790900 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e72fda1743fc4f96b29add6e36e884de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e72fda1743fc4f96b29add6e36e884de already in progress: creating tablet
I20250112 20:51:34.791903 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eb174dc18b7d43d69147c2d69b5f458d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.792382 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb174dc18b7d43d69147c2d69b5f458d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.793269 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.793798 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_75b794cc68ca4a2c83131bcb146d3b66" } new_table_name: "default.Table_e907fd649a434fe9b3a51b2619964cb3" modify_external_catalogs: true
I20250112 20:51:34.793920 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.794574 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d5269a5c8c4c049265e18bff4e2c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.794871 882 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5af0bc83bc415b972cbbe0fb39f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.795020 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.795116 885 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.795455 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.795816 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.796346 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 with delete_type TABLET_DATA_DELETED (replaced by tablet d570daec845f40778c48ff3f2e2e30cb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.796917 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c14102aecb743e2a82660039bd0f038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:34.804986 1974 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:34.807154 983 consensus_peers.cc:487] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c3f359655c94f1d905943069548845b. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.808455 1816 webserver.cc:458] Webserver started at http://127.0.104.60:44167/ using document root <none> and password file <none>
I20250112 20:51:34.810195 1164 consensus_queue.cc:260] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.814079 1163 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.814314 1163 ts_tablet_manager.cc:1397] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.153s user 0.015s sys 0.019s
I20250112 20:51:34.815208 1163 raft_consensus.cc:357] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.815557 1163 raft_consensus.cc:383] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.815721 1163 raft_consensus.cc:738] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.816035 1163 consensus_queue.cc:260] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.807053 1858 raft_consensus.cc:491] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.818778 1858 raft_consensus.cc:513] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.818915 1163 ts_tablet_manager.cc:1428] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:34.811553 1178 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.819356 1163 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.819357 1178 ts_tablet_manager.cc:1397] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.167s user 0.002s sys 0.002s
I20250112 20:51:34.820169 1858 leader_election.cc:290] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.820137 1178 raft_consensus.cc:357] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.822069 1178 raft_consensus.cc:383] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.822182 1178 raft_consensus.cc:738] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.822467 1178 consensus_queue.cc:260] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.822503 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c13126e53b46c0a1e7beafe1b0d295" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.823037 1178 ts_tablet_manager.cc:1428] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:34.823912 1975 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:34.824688 847 leader_election.cc:343] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.825699 1164 ts_tablet_manager.cc:1428] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20250112 20:51:34.826349 1178 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:34.810000 1816 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:34.827790 1816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:34.829941 1164 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.820814 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c13126e53b46c0a1e7beafe1b0d295" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.840224 1821 consensus_queue.cc:1035] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.843205 1984 tablet.cc:1721] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:34.843475 1184 ts_tablet_manager.cc:1905] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.843370 1048 raft_consensus.cc:2463] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:34.844405 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 725e56fa8ac343e5b1ec1214da02b7d4 has no consensus state
I20250112 20:51:34.847236 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_588580b3605944d0b5434d8b59053fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.847812 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_588580b3605944d0b5434d8b59053fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.850764 1184 ts_tablet_manager.cc:1918] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 20:51:34.850940 847 leader_election.cc:304] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.851797 1983 raft_consensus.cc:2798] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.852347 1983 raft_consensus.cc:491] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.852653 1983 raft_consensus.cc:3054] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:34.854025 847 consensus_peers.cc:487] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00c0712ab2de43eea8c77ab39a971a2c. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:34.855374 1942 raft_consensus.cc:491] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.855484 1983 raft_consensus.cc:513] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.855525 1942 raft_consensus.cc:513] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.856169 1983 leader_election.cc:290] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.856448 1186 ts_tablet_manager.cc:1918] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.856590 1186 log.cc:1198] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8c38c8594a384689ae57c26174765284
I20250112 20:51:34.857506 1942 leader_election.cc:290] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.857864 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c13126e53b46c0a1e7beafe1b0d295" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
W20250112 20:51:34.858481 847 leader_election.cc:343] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.858729 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590f9615c4d47abaad1a36ac74b3c80" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:34.858951 774 raft_consensus.cc:2463] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:34.859323 847 leader_election.cc:304] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:34.859622 1942 raft_consensus.cc:2798] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.859786 1942 raft_consensus.cc:491] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.859941 1942 raft_consensus.cc:3054] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.860026 1178 tablet_bootstrap.cc:654] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.861529 1186 ts_tablet_manager.cc:1939] T 8c38c8594a384689ae57c26174765284 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.864208 1942 raft_consensus.cc:513] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.864634 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 725e56fa8ac343e5b1ec1214da02b7d4 (table default.Table_be8270d1ca354e748dac253a4aeb09a3 [id=a63f77f9d86346d0aed333ccca1a28ce], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 725e56fa8ac343e5b1ec1214da02b7d4 has no consensus state
I20250112 20:51:34.865048 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 335) took 2475 ms. Trace:
I20250112 20:51:34.865165 1186 rpcz_store.cc:276] 0112 20:51:32.389611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.389662 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:34.770538 (+2380876us) ts_tablet_manager.cc:1145] Deleting tablet 8c38c8594a384689ae57c26174765284
0112 20:51:34.770549 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.814681 (+ 44132us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.840662 (+ 25981us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.864991 (+ 24329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380712}
I20250112 20:51:34.866024 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 345) took 2444 ms. Trace:
I20250112 20:51:34.866127 1186 rpcz_store.cc:276] 0112 20:51:32.421960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.422011 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:34.865852 (+2443841us) ts_tablet_manager.cc:1145] Deleting tablet 8d08048c402345a686d1e3fb3039bbc1
0112 20:51:34.865870 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.865964 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":2443345}
I20250112 20:51:34.866492 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 347) took 2437 ms. Trace:
I20250112 20:51:34.866603 1186 rpcz_store.cc:276] 0112 20:51:32.428837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.428879 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:34.866359 (+2437480us) ts_tablet_manager.cc:1145] Deleting tablet ad1a4b5ac2f94874b99b3ef790a034fd
0112 20:51:34.866368 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.866463 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419156}
I20250112 20:51:34.866748 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8c38c8594a384689ae57c26174765284 (table default.Table_a078ad9ead144747a9f5af58eb338733 [id=54146c6899f04289a61d6f84ce1c4250]) successfully deleted
W20250112 20:51:34.867323 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8d08048c402345a686d1e3fb3039bbc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d08048c402345a686d1e3fb3039bbc1 already in progress: creating tablet
W20250112 20:51:34.867663 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ad1a4b5ac2f94874b99b3ef790a034fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad1a4b5ac2f94874b99b3ef790a034fd already in progress: creating tablet
I20250112 20:51:34.868062 1942 leader_election.cc:290] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.868248 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8c9a7a06fb384d4aa61879e57247d36e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.868800 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c9a7a06fb384d4aa61879e57247d36e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.868995 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590f9615c4d47abaad1a36ac74b3c80" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:34.869186 770 raft_consensus.cc:3054] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.872905 1186 tablet_replica.cc:331] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.873123 1186 raft_consensus.cc:2238] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.873281 1186 raft_consensus.cc:2267] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.857046 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c13126e53b46c0a1e7beafe1b0d295" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.873581 1048 raft_consensus.cc:3054] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:34.874085 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d167065029a46b791d413db8a2c4bcc has no consensus state
I20250112 20:51:34.874361 455 master_service.cc:946] ReplaceTablet: received request to replace tablet aa9219fc971c4d15b0d383b20b61e540 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.858166 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590f9615c4d47abaad1a36ac74b3c80" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:34.877079 1048 raft_consensus.cc:2463] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.867318 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e590f9615c4d47abaad1a36ac74b3c80" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:34.877382 1184 log.cc:1198] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5f012db68caf4d7d86afa6deb9f2ffaa
I20250112 20:51:34.877336 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9ced728bcaeb4a668b3c1cc24916d012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.877933 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ced728bcaeb4a668b3c1cc24916d012 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.878827 1186 ts_tablet_manager.cc:1905] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:34.880417 847 leader_election.cc:343] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.882033 847 leader_election.cc:304] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.883249 1184 ts_tablet_manager.cc:1939] T 5f012db68caf4d7d86afa6deb9f2ffaa P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.883601 1163 tablet_bootstrap.cc:654] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:34.884070 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22f6b2c4ea5f4167896c78b1c64cbb62 consensus state has no leader
I20250112 20:51:34.884384 448 master_service.cc:946] ReplaceTablet: received request to replace tablet d7117edf078848c0a1c83ab8e804dfed from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.886590 1163 tablet_bootstrap.cc:492] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.886795 1164 tablet_bootstrap.cc:654] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.887272 1163 ts_tablet_manager.cc:1397] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20250112 20:51:34.888384 1163 raft_consensus.cc:357] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.891304 1185 tablet_replica.cc:331] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:34.892885 847 leader_election.cc:343] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.893198 1982 raft_consensus.cc:2798] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:34.893209 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b37b00c49f234dc89d63f644febc23fb has no consensus state
I20250112 20:51:34.893342 1982 raft_consensus.cc:695] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.893499 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.893743 1982 consensus_queue.cc:237] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.894191 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c06477d484e0483589b0530cf5787b9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:34.894716 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c06477d484e0483589b0530cf5787b9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:34.895573 770 raft_consensus.cc:2463] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:34.888854 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 387) took 1656 ms. Trace:
I20250112 20:51:34.896356 1184 rpcz_store.cc:276] 0112 20:51:33.232802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.233707 (+ 905us) service_pool.cc:225] Handling call
0112 20:51:34.789703 (+1555996us) ts_tablet_manager.cc:1145] Deleting tablet 5f012db68caf4d7d86afa6deb9f2ffaa
0112 20:51:34.789712 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.847950 (+ 58238us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.850663 (+ 2713us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.888794 (+ 38131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540048}
I20250112 20:51:34.896566 847 leader_election.cc:304] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:34.896781 1185 raft_consensus.cc:2238] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.896821 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_a77dee800e62415db3cada3902c1efcc [id=9448c50d601f435e81de18e95f86efb0]) successfully deleted
I20250112 20:51:34.897159 1185 raft_consensus.cc:2267] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.897241 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 385) took 1685 ms. Trace:
I20250112 20:51:34.897396 1184 rpcz_store.cc:276] 0112 20:51:33.211776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.219372 (+ 7596us) service_pool.cc:225] Handling call
0112 20:51:34.897070 (+1677698us) ts_tablet_manager.cc:1145] Deleting tablet 3bbb679585fc44699553fe85ea044566
0112 20:51:34.897081 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.897191 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638479}
I20250112 20:51:34.897922 1942 raft_consensus.cc:2798] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:34.897935 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3bbb679585fc44699553fe85ea044566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb679585fc44699553fe85ea044566 already in progress: creating tablet
I20250112 20:51:34.896155 1821 raft_consensus.cc:491] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.898162 1942 raft_consensus.cc:695] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:34.898247 1184 tablet_replica.cc:331] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.898159 1821 raft_consensus.cc:513] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.898439 1184 raft_consensus.cc:2238] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.898866 1821 leader_election.cc:290] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.899000 1184 raft_consensus.cc:2267] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.898599 1942 consensus_queue.cc:237] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.900046 1184 ts_tablet_manager.cc:1905] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.901074 1185 ts_tablet_manager.cc:1905] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.901212 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.901418 911 raft_consensus.cc:2463] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:34.901937 983 leader_election.cc:304] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:34.902319 1821 raft_consensus.cc:2798] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:34.902423 1821 raft_consensus.cc:491] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:34.902503 1821 raft_consensus.cc:3054] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.904186 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:34.904817 983 leader_election.cc:343] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.905269 1186 ts_tablet_manager.cc:1918] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.905437 1186 log.cc:1198] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:34.905354 1821 raft_consensus.cc:513] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.905841 1186 ts_tablet_manager.cc:1939] T d770e0399e80491f8de9b08d1eb2f139 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.906050 1821 leader_election.cc:290] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:34.907239 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
W20250112 20:51:34.907757 983 leader_election.cc:343] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.908692 1164 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:34.908933 1164 ts_tablet_manager.cc:1397] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20250112 20:51:34.909389 1163 raft_consensus.cc:383] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.909634 1163 raft_consensus.cc:738] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.910089 1163 consensus_queue.cc:260] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.911783 1931 raft_consensus.cc:491] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:34.911949 1931 raft_consensus.cc:513] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:34.912500 1931 leader_election.cc:290] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:34.913149 1163 ts_tablet_manager.cc:1428] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250112 20:51:34.913475 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:34.914012 710 leader_election.cc:343] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.909773 1164 raft_consensus.cc:357] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.914449 1164 raft_consensus.cc:383] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.914541 1164 raft_consensus.cc:738] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.915082 1164 consensus_queue.cc:260] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.917760 1163 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.919447 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:34.919648 1185 ts_tablet_manager.cc:1918] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.919768 1185 log.cc:1198] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f769533bfde0448792d01b883e41763c
I20250112 20:51:34.920451 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4629339765453d988953799095b3ac" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:34.920763 911 raft_consensus.cc:3054] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:34.921051 1163 tablet_bootstrap.cc:654] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.923409 911 raft_consensus.cc:2463] T bc4629339765453d988953799095b3ac P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:34.918673 1184 ts_tablet_manager.cc:1918] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:34.923918 983 leader_election.cc:304] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:34.924341 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 348) took 2479 ms. Trace:
I20250112 20:51:34.924432 1186 rpcz_store.cc:276] 0112 20:51:32.445265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.445325 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:34.872829 (+2427504us) ts_tablet_manager.cc:1145] Deleting tablet d770e0399e80491f8de9b08d1eb2f139
0112 20:51:34.872840 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.894168 (+ 21328us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.905155 (+ 10987us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.924282 (+ 19127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424209}
I20250112 20:51:34.924827 1186 tablet_replica.cc:331] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.925022 1186 raft_consensus.cc:2238] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.925953 1186 raft_consensus.cc:2267] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.926512 1186 ts_tablet_manager.cc:1905] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.927067 1821 raft_consensus.cc:2798] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:34.927196 1821 raft_consensus.cc:695] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:34.927445 1821 consensus_queue.cc:237] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:34.920089 710 leader_election.cc:343] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:34.928911 710 leader_election.cc:304] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:34.929426 1931 raft_consensus.cc:2743] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:34.930128 1185 ts_tablet_manager.cc:1939] T f769533bfde0448792d01b883e41763c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.931085 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 364) took 2265 ms. Trace:
I20250112 20:51:34.931697 1185 rpcz_store.cc:276] 0112 20:51:32.665942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.680079 (+ 14137us) service_pool.cc:225] Handling call
0112 20:51:34.889529 (+2209450us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:34.889542 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.904827 (+ 15285us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.919551 (+ 14724us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.931022 (+ 11471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2209278}
I20250112 20:51:34.932161 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f769533bfde0448792d01b883e41763c (table default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]) successfully deleted
I20250112 20:51:34.929432 1164 ts_tablet_manager.cc:1428] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250112 20:51:34.933053 1185 tablet_replica.cc:331] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:34.934495 1185 raft_consensus.cc:2238] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.934715 1185 raft_consensus.cc:2267] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.935310 1185 ts_tablet_manager.cc:1905] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.935918 1816 fs_manager.cc:714] Time spent opening directory manager: real 0.101s user 0.006s sys 0.005s
I20250112 20:51:34.936820 1186 ts_tablet_manager.cc:1918] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.936952 1186 log.cc:1198] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/00c0712ab2de43eea8c77ab39a971a2c
I20250112 20:51:34.937443 1186 ts_tablet_manager.cc:1939] T 00c0712ab2de43eea8c77ab39a971a2c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.938242 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 350) took 2489 ms. Trace:
I20250112 20:51:34.938390 1186 rpcz_store.cc:276] 0112 20:51:32.448320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.448366 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:34.924710 (+2476344us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:34.924720 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.933239 (+ 8519us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.936678 (+ 3439us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.938205 (+ 1527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2475170}
I20250112 20:51:34.938771 1186 tablet_replica.cc:331] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.938958 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 00c0712ab2de43eea8c77ab39a971a2c (table default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]) successfully deleted
I20250112 20:51:34.938999 1186 raft_consensus.cc:2238] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.939145 1186 raft_consensus.cc:2267] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.938565 1184 log.cc:1198] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5ca061b82a894e91b9508613ccd8ecd7
I20250112 20:51:34.939288 1931 consensus_queue.cc:1035] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.939841 1184 ts_tablet_manager.cc:1939] T 5ca061b82a894e91b9508613ccd8ecd7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.934312 1164 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:34.943604 1164 tablet_bootstrap.cc:654] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.944856 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 384) took 1738 ms. Trace:
I20250112 20:51:34.945482 1185 ts_tablet_manager.cc:1918] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:34.945907 1184 rpcz_store.cc:276] 0112 20:51:33.206863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.208605 (+ 1742us) service_pool.cc:225] Handling call
0112 20:51:34.898164 (+1689559us) ts_tablet_manager.cc:1145] Deleting tablet 5ca061b82a894e91b9508613ccd8ecd7
0112 20:51:34.898175 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.903362 (+ 5187us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.918544 (+ 15182us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.944815 (+ 26271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638403}
I20250112 20:51:34.946520 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]) successfully deleted
I20250112 20:51:34.947010 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 389) took 1496 ms. Trace:
I20250112 20:51:34.947892 1186 ts_tablet_manager.cc:1905] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.948820 1185 log.cc:1198] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3c14102aecb743e2a82660039bd0f038
I20250112 20:51:34.949445 1185 ts_tablet_manager.cc:1939] T 3c14102aecb743e2a82660039bd0f038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.950649 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 365) took 2284 ms. Trace:
I20250112 20:51:34.952941 1163 tablet_bootstrap.cc:492] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:34.953331 1163 ts_tablet_manager.cc:1397] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20250112 20:51:34.953055 1184 rpcz_store.cc:276] 0112 20:51:33.450272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.450340 (+ 68us) service_pool.cc:225] Handling call
0112 20:51:34.946843 (+1496503us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:34.946853 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.946975 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495413}
I20250112 20:51:34.954147 1185 rpcz_store.cc:276] 0112 20:51:32.666028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.680256 (+ 14228us) service_pool.cc:225] Handling call
0112 20:51:34.932696 (+2252440us) ts_tablet_manager.cc:1145] Deleting tablet 3c14102aecb743e2a82660039bd0f038
0112 20:51:34.932969 (+ 273us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.938396 (+ 5427us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.945343 (+ 6947us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.950607 (+ 5264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252305}
W20250112 20:51:34.954628 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4d29316b9d2441bca97f56bdb0e56f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d29316b9d2441bca97f56bdb0e56f0d already in progress: creating tablet
I20250112 20:51:34.955065 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3c14102aecb743e2a82660039bd0f038 (table default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]) successfully deleted
I20250112 20:51:34.955009 1163 raft_consensus.cc:357] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.955245 1163 raft_consensus.cc:383] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:34.955451 1184 tablet_replica.cc:331] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:34.955578 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 367) took 2289 ms. Trace:
I20250112 20:51:34.955973 1163 raft_consensus.cc:738] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:34.956100 1185 rpcz_store.cc:276] 0112 20:51:32.666199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.683638 (+ 17439us) service_pool.cc:225] Handling call
0112 20:51:34.955420 (+2271782us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:34.955431 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.955545 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2271521}
I20250112 20:51:34.956427 1184 raft_consensus.cc:2238] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.956851 1184 raft_consensus.cc:2267] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.956961 1186 ts_tablet_manager.cc:1918] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:34.957022 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3343e1d7f1d04a498bd0ee00311b6a61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3343e1d7f1d04a498bd0ee00311b6a61 already in progress: creating tablet
I20250112 20:51:34.957096 1186 log.cc:1198] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4c3f359655c94f1d905943069548845b
I20250112 20:51:34.957486 1186 ts_tablet_manager.cc:1939] T 4c3f359655c94f1d905943069548845b P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.957525 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 368) took 2291 ms. Trace:
I20250112 20:51:34.957942 1184 ts_tablet_manager.cc:1905] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.958252 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 351) took 2508 ms. Trace:
I20250112 20:51:34.958359 1186 rpcz_store.cc:276] 0112 20:51:32.449706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.449774 (+ 68us) service_pool.cc:225] Handling call
0112 20:51:34.938695 (+2488921us) ts_tablet_manager.cc:1145] Deleting tablet 4c3f359655c94f1d905943069548845b
0112 20:51:34.938707 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.953041 (+ 14334us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.956857 (+ 3816us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.958200 (+ 1343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2483626}
I20250112 20:51:34.958456 1163 consensus_queue.cc:260] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:34.958719 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4c3f359655c94f1d905943069548845b (table default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]) successfully deleted
I20250112 20:51:34.958891 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 371) took 2318 ms. Trace:
I20250112 20:51:34.958982 1186 rpcz_store.cc:276] 0112 20:51:32.640716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.648860 (+ 8144us) service_pool.cc:225] Handling call
0112 20:51:34.958728 (+2309868us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:34.958746 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.958840 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309657}
I20250112 20:51:34.959156 1185 rpcz_store.cc:276] 0112 20:51:32.666279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.683917 (+ 17638us) service_pool.cc:225] Handling call
0112 20:51:34.957356 (+2273439us) ts_tablet_manager.cc:1145] Deleting tablet d570daec845f40778c48ff3f2e2e30cb
0112 20:51:34.957369 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.957481 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272931}
W20250112 20:51:34.959532 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3343e1d7f1d04a498bd0ee00311b6a61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3343e1d7f1d04a498bd0ee00311b6a61 already in progress: creating tablet
I20250112 20:51:34.959625 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 372) took 2318 ms. Trace:
I20250112 20:51:34.959708 1186 rpcz_store.cc:276] 0112 20:51:32.640839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.649101 (+ 8262us) service_pool.cc:225] Handling call
0112 20:51:34.959462 (+2310361us) ts_tablet_manager.cc:1145] Deleting tablet d570daec845f40778c48ff3f2e2e30cb
0112 20:51:34.959469 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.959574 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310205}
W20250112 20:51:34.959873 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d570daec845f40778c48ff3f2e2e30cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d570daec845f40778c48ff3f2e2e30cb already in progress: creating tablet
W20250112 20:51:34.960140 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d570daec845f40778c48ff3f2e2e30cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d570daec845f40778c48ff3f2e2e30cb already in progress: creating tablet
I20250112 20:51:34.960140 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 373) took 2051 ms. Trace:
I20250112 20:51:34.960228 1186 rpcz_store.cc:276] 0112 20:51:32.908894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.909252 (+ 358us) service_pool.cc:225] Handling call
0112 20:51:34.960009 (+2050757us) ts_tablet_manager.cc:1145] Deleting tablet 42eb0f79df5444f8a8360921d5f2b9ee
0112 20:51:34.960017 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.960088 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2050450}
W20250112 20:51:34.960530 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 42eb0f79df5444f8a8360921d5f2b9ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42eb0f79df5444f8a8360921d5f2b9ee already in progress: creating tablet
I20250112 20:51:34.960594 1186 tablet_replica.cc:331] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.960811 1186 raft_consensus.cc:2238] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.961092 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 369) took 2211 ms. Trace:
I20250112 20:51:34.961230 1185 rpcz_store.cc:276] 0112 20:51:32.749487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.749613 (+ 126us) service_pool.cc:225] Handling call
0112 20:51:34.960908 (+2211295us) ts_tablet_manager.cc:1145] Deleting tablet 42eb0f79df5444f8a8360921d5f2b9ee
0112 20:51:34.960917 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.961050 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197823}
W20250112 20:51:34.961637 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 42eb0f79df5444f8a8360921d5f2b9ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42eb0f79df5444f8a8360921d5f2b9ee already in progress: creating tablet
I20250112 20:51:34.962186 1186 raft_consensus.cc:2267] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.962725 1186 ts_tablet_manager.cc:1905] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.963773 1163 ts_tablet_manager.cc:1428] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250112 20:51:34.966023 1163 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:34.968919 1186 ts_tablet_manager.cc:1918] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:34.969048 1186 log.cc:1198] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/44f11304909b416aa1e9db1abf633a89
I20250112 20:51:34.969259 1185 tablet_replica.cc:331] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:34.969599 1186 ts_tablet_manager.cc:1939] T 44f11304909b416aa1e9db1abf633a89 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.969877 1185 raft_consensus.cc:2238] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:34.970325 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 375) took 2046 ms. Trace:
I20250112 20:51:34.970440 1186 rpcz_store.cc:276] 0112 20:51:32.923456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.924470 (+ 1014us) service_pool.cc:225] Handling call
0112 20:51:34.960523 (+2036053us) ts_tablet_manager.cc:1145] Deleting tablet 44f11304909b416aa1e9db1abf633a89
0112 20:51:34.960531 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.966081 (+ 5550us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.968804 (+ 2723us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.970294 (+ 1490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035843}
I20250112 20:51:34.970356 1185 raft_consensus.cc:2267] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.970824 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 378) took 2039 ms. Trace:
I20250112 20:51:34.970921 1186 rpcz_store.cc:276] 0112 20:51:32.931558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.931615 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:34.970691 (+2039076us) ts_tablet_manager.cc:1145] Deleting tablet 7deb6e37f75e4aebacb04c4553049940
0112 20:51:34.970701 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.970797 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2038889}
I20250112 20:51:34.971033 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]) successfully deleted
I20250112 20:51:34.971259 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 380) took 2030 ms. Trace:
I20250112 20:51:34.971352 1186 rpcz_store.cc:276] 0112 20:51:32.941088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.941159 (+ 71us) service_pool.cc:225] Handling call
0112 20:51:34.971128 (+2029969us) ts_tablet_manager.cc:1145] Deleting tablet 8f636df32bd34438bb54989736eaa431
0112 20:51:34.971136 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.971229 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029675}
I20250112 20:51:34.971585 1186 tablet_replica.cc:331] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.971792 1186 raft_consensus.cc:2238] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:34.971803 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8f636df32bd34438bb54989736eaa431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f636df32bd34438bb54989736eaa431 already in progress: creating tablet
I20250112 20:51:34.972100 1186 raft_consensus.cc:2267] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.972450 1185 ts_tablet_manager.cc:1905] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.972620 1186 ts_tablet_manager.cc:1905] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.978257 1163 tablet_bootstrap.cc:654] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:34.978679 1186 ts_tablet_manager.cc:1918] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:34.978866 1186 log.cc:1198] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5ca061b82a894e91b9508613ccd8ecd7
I20250112 20:51:34.979269 1186 ts_tablet_manager.cc:1939] T 5ca061b82a894e91b9508613ccd8ecd7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.980067 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 391) took 1758 ms. Trace:
I20250112 20:51:34.980160 1186 rpcz_store.cc:276] 0112 20:51:33.221972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.222051 (+ 79us) service_pool.cc:225] Handling call
0112 20:51:34.971532 (+1749481us) ts_tablet_manager.cc:1145] Deleting tablet 5ca061b82a894e91b9508613ccd8ecd7
0112 20:51:34.971539 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.975586 (+ 4047us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.978579 (+ 2993us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.980015 (+ 1436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739782}
I20250112 20:51:34.980572 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 392) took 1753 ms. Trace:
I20250112 20:51:34.980631 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]) successfully deleted
I20250112 20:51:34.980664 1186 rpcz_store.cc:276] 0112 20:51:33.227421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.227493 (+ 72us) service_pool.cc:225] Handling call
0112 20:51:34.980428 (+1752935us) ts_tablet_manager.cc:1145] Deleting tablet 3bbb679585fc44699553fe85ea044566
0112 20:51:34.980437 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.980527 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746016}
I20250112 20:51:34.981036 1186 tablet_replica.cc:331] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:34.981204 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3bbb679585fc44699553fe85ea044566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb679585fc44699553fe85ea044566 already in progress: creating tablet
I20250112 20:51:34.981271 1186 raft_consensus.cc:2238] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.981892 1186 raft_consensus.cc:2267] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:34.982393 1186 ts_tablet_manager.cc:1905] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.983012 1184 ts_tablet_manager.cc:1918] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:34.988595 1186 ts_tablet_manager.cc:1918] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 20:51:34.988721 1186 log.cc:1198] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5f012db68caf4d7d86afa6deb9f2ffaa
I20250112 20:51:34.989238 1186 ts_tablet_manager.cc:1939] T 5f012db68caf4d7d86afa6deb9f2ffaa P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:34.989364 1184 log.cc:1198] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/a48df2ec556d43a3b22129b1177ebf74
I20250112 20:51:34.990044 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 395) took 1754 ms. Trace:
I20250112 20:51:34.990163 1186 rpcz_store.cc:276] 0112 20:51:33.235940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.236290 (+ 350us) service_pool.cc:225] Handling call
0112 20:51:34.980926 (+1744636us) ts_tablet_manager.cc:1145] Deleting tablet 5f012db68caf4d7d86afa6deb9f2ffaa
0112 20:51:34.980934 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.985409 (+ 4475us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.988499 (+ 3090us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.990010 (+ 1511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1706682}
I20250112 20:51:34.990633 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_a77dee800e62415db3cada3902c1efcc [id=9448c50d601f435e81de18e95f86efb0]) successfully deleted
I20250112 20:51:34.990099 1184 ts_tablet_manager.cc:1939] T a48df2ec556d43a3b22129b1177ebf74 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:34.991107 1186 tablet_replica.cc:331] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:34.991201 1185 ts_tablet_manager.cc:1918] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:34.991333 1186 raft_consensus.cc:2238] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:34.991681 1186 raft_consensus.cc:2267] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:34.992249 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39271, user_credentials={real_user=slave}} blocked reactor thread for 61573.6us
I20250112 20:51:34.992949 1186 ts_tablet_manager.cc:1905] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:34.993538 1185 log.cc:1198] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/44f11304909b416aa1e9db1abf633a89
I20250112 20:51:34.994449 1178 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:34.994488 1821 consensus_queue.cc:1035] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:34.994606 1178 ts_tablet_manager.cc:1397] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.168s user 0.003s sys 0.003s
I20250112 20:51:34.994746 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d97ac93467a4a7a8ae44d52719f191c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:34.995596 1185 ts_tablet_manager.cc:1939] T 44f11304909b416aa1e9db1abf633a89 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:34.996701 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 370) took 2217 ms. Trace:
I20250112 20:51:34.999944 1185 rpcz_store.cc:276] 0112 20:51:32.778924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.779233 (+ 309us) service_pool.cc:225] Handling call
0112 20:51:34.969191 (+2189958us) ts_tablet_manager.cc:1145] Deleting tablet 44f11304909b416aa1e9db1abf633a89
0112 20:51:34.969206 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.982361 (+ 13155us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.991093 (+ 8732us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.996641 (+ 5548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147480}
I20250112 20:51:35.000470 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 44f11304909b416aa1e9db1abf633a89 (table default.Table_b0787900a74e4ae79083035c64b84c00 [id=14ecb9204295481eb572f350eb539e4e]) successfully deleted
I20250112 20:51:35.002097 1178 raft_consensus.cc:357] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.002306 1178 raft_consensus.cc:383] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.002398 1178 raft_consensus.cc:738] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.002619 1178 consensus_queue.cc:260] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.002734 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 373) took 2110 ms. Trace:
I20250112 20:51:35.003032 1185 rpcz_store.cc:276] 0112 20:51:32.892491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.893258 (+ 767us) service_pool.cc:225] Handling call
0112 20:51:35.002561 (+2109303us) ts_tablet_manager.cc:1145] Deleting tablet 8f636df32bd34438bb54989736eaa431
0112 20:51:35.002573 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.002683 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108365}
W20250112 20:51:35.003599 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8f636df32bd34438bb54989736eaa431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f636df32bd34438bb54989736eaa431 already in progress: creating tablet
I20250112 20:51:35.004585 1164 tablet_bootstrap.cc:492] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.006706 1178 ts_tablet_manager.cc:1428] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:35.007141 1998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:35.008708 1178 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.009274 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11d93ce9bed540a6b14e29a387b232ba" } new_table_name: "default.Table_47d0e6446c184df28dce16b823202a5b" modify_external_catalogs: true
I20250112 20:51:35.010970 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15e3fd8e71984a81acb2dabf8f3563f7" } modify_external_catalogs: true
I20250112 20:51:35.011168 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_15e3fd8e71984a81acb2dabf8f3563f7" } modify_external_catalogs: true
I20250112 20:51:35.014202 1186 ts_tablet_manager.cc:1918] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.014346 1186 log.cc:1198] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:35.014725 1186 ts_tablet_manager.cc:1939] T e72fda1743fc4f96b29add6e36e884de P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.017206 455 master_service.cc:946] ReplaceTablet: received request to replace tablet a3ca4d49496e42ddb263264ceef2f27a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.019553 1816 fs_manager.cc:730] Time spent opening block manager: real 0.074s user 0.006s sys 0.003s
I20250112 20:51:35.020973 1164 ts_tablet_manager.cc:1397] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.007s
I20250112 20:51:35.021816 1164 raft_consensus.cc:357] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.022024 1164 raft_consensus.cc:383] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.022133 1164 raft_consensus.cc:738] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.022397 1164 consensus_queue.cc:260] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.023032 1164 ts_tablet_manager.cc:1428] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:35.023330 1164 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.025182 1164 tablet_bootstrap.cc:654] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.026072 741 tablet_service.cc:1467] Processing CreateTablet for tablet d17e6d263014448b96c5d1f03013e6e7 (DEFAULT_TABLE table=default.Table_a1b5b3029a0f4295904392987db19062 [id=04bd0f25c5db41039b53f79ce61124ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.026456 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17e6d263014448b96c5d1f03013e6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.026435 886 tablet_service.cc:1467] Processing CreateTablet for tablet d17e6d263014448b96c5d1f03013e6e7 (DEFAULT_TABLE table=default.Table_a1b5b3029a0f4295904392987db19062 [id=04bd0f25c5db41039b53f79ce61124ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.026790 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17e6d263014448b96c5d1f03013e6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.027328 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:35.028317 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 372) took 2135 ms. Trace:
I20250112 20:51:35.028421 1185 rpcz_store.cc:276] 0112 20:51:32.892386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:32.893082 (+ 696us) service_pool.cc:225] Handling call
0112 20:51:35.003885 (+2110803us) ts_tablet_manager.cc:1145] Deleting tablet 7deb6e37f75e4aebacb04c4553049940
0112 20:51:35.003895 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.028252 (+ 24357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108442}
I20250112 20:51:35.028885 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 385) took 1871 ms. Trace:
I20250112 20:51:35.028977 1185 rpcz_store.cc:276] 0112 20:51:33.157151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.157204 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:35.028668 (+1871464us) ts_tablet_manager.cc:1145] Deleting tablet 3bbb679585fc44699553fe85ea044566
0112 20:51:35.028677 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.028824 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871219}
I20250112 20:51:35.029106 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bb77de2d74804434a45865840ea7eea0" } modify_external_catalogs: true
I20250112 20:51:35.029268 1185 tablet_replica.cc:331] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.029294 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bb77de2d74804434a45865840ea7eea0" } modify_external_catalogs: true
I20250112 20:51:35.029036 1019 tablet_service.cc:1467] Processing CreateTablet for tablet d17e6d263014448b96c5d1f03013e6e7 (DEFAULT_TABLE table=default.Table_a1b5b3029a0f4295904392987db19062 [id=04bd0f25c5db41039b53f79ce61124ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.029464 1185 raft_consensus.cc:2238] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.029695 1185 raft_consensus.cc:2267] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.030233 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17e6d263014448b96c5d1f03013e6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.030230 1185 ts_tablet_manager.cc:1905] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.030586 752 tablet_service.cc:1467] Processing CreateTablet for tablet d72ae28879dc45429de2ab0d66276255 (DEFAULT_TABLE table=default.Table_7f164d4ff332493e885b6ef5272e508a [id=9a4f3120210440ea957d92ed21582da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.030684 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 24a5162a25ce47519a091742f3d5ca31 (DEFAULT_TABLE table=default.Table_7f122343e2d74bf7b3eefea9de3ff725 [id=02dcec3eeef04e78a9c707989a8578bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.030753 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1b5b3029a0f4295904392987db19062" } modify_external_catalogs: true
I20250112 20:51:35.030900 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1b5b3029a0f4295904392987db19062" } modify_external_catalogs: true
I20250112 20:51:35.030964 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ae28879dc45429de2ab0d66276255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.032938 745 tablet_service.cc:1467] Processing CreateTablet for tablet 24a5162a25ce47519a091742f3d5ca31 (DEFAULT_TABLE table=default.Table_7f122343e2d74bf7b3eefea9de3ff725 [id=02dcec3eeef04e78a9c707989a8578bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.033041 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a5162a25ce47519a091742f3d5ca31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.033025 885 tablet_service.cc:1467] Processing CreateTablet for tablet 24a5162a25ce47519a091742f3d5ca31 (DEFAULT_TABLE table=default.Table_7f122343e2d74bf7b3eefea9de3ff725 [id=02dcec3eeef04e78a9c707989a8578bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.033272 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a5162a25ce47519a091742f3d5ca31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.033391 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24a5162a25ce47519a091742f3d5ca31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.033581 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d72ae28879dc45429de2ab0d66276255 (DEFAULT_TABLE table=default.Table_7f164d4ff332493e885b6ef5272e508a [id=9a4f3120210440ea957d92ed21582da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.033944 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ae28879dc45429de2ab0d66276255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.039083 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:35.039345 446 catalog_manager.cc:5526] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:35.046485 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc90d3722fd14d59b9994655cd30ef2b consensus state has no leader
W20250112 20:51:35.048085 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3bbb679585fc44699553fe85ea044566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb679585fc44699553fe85ea044566 already in progress: creating tablet
I20250112 20:51:35.050187 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet af1aac31fd32400b9094f8e2de4fe42c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.019801 1816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
uuid: "0f5c09e899184de6b770dca873501062"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:35.055455 1816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:35.056547 1185 ts_tablet_manager.cc:1918] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:35.056710 1185 log.cc:1198] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5ca061b82a894e91b9508613ccd8ecd7
I20250112 20:51:35.057538 884 tablet_service.cc:1467] Processing CreateTablet for tablet b37b00c49f234dc89d63f644febc23fb (DEFAULT_TABLE table=default.Table_e907fd649a434fe9b3a51b2619964cb3 [id=8997a20696974e4fb75b4f18903d7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.057822 882 tablet_service.cc:1467] Processing CreateTablet for tablet 65784493c6ef417993b1f6371cab47d6 (DEFAULT_TABLE table=default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 [id=4130b8a93c4645839e4fad41a2127529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.057968 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37b00c49f234dc89d63f644febc23fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.058203 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65784493c6ef417993b1f6371cab47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.060925 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 393) took 1827 ms. Trace:
I20250112 20:51:35.061094 874 tablet_service.cc:1467] Processing CreateTablet for tablet a20d85a4be874b0b90c528a631a29aed (DEFAULT_TABLE table=default.Table_ec3a27d13db349b4a78c058fdea0ad2d [id=c628277a9fdf4ff7b2df5f322a8f5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.061406 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20d85a4be874b0b90c528a631a29aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.061362 1011 tablet_service.cc:1467] Processing CreateTablet for tablet b37b00c49f234dc89d63f644febc23fb (DEFAULT_TABLE table=default.Table_e907fd649a434fe9b3a51b2619964cb3 [id=8997a20696974e4fb75b4f18903d7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.061834 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37b00c49f234dc89d63f644febc23fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.066428 877 tablet_service.cc:1467] Processing CreateTablet for tablet 615d75bc22df45cc876788a266e983cc (DEFAULT_TABLE table=default.Table_b5d6bc8b522a4be197bd045c77d4a5f4 [id=3066f1ca47fe4a39841cfd5b6028f16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.066785 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615d75bc22df45cc876788a266e983cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.067569 886 tablet_service.cc:1467] Processing CreateTablet for tablet d72ae28879dc45429de2ab0d66276255 (DEFAULT_TABLE table=default.Table_7f164d4ff332493e885b6ef5272e508a [id=9a4f3120210440ea957d92ed21582da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.068558 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ae28879dc45429de2ab0d66276255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.061031 1186 rpcz_store.cc:276] 0112 20:51:33.233061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.233138 (+ 77us) service_pool.cc:225] Handling call
0112 20:51:34.991041 (+1757903us) ts_tablet_manager.cc:1145] Deleting tablet e72fda1743fc4f96b29add6e36e884de
0112 20:51:34.991049 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.011156 (+ 20107us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.014080 (+ 2924us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.060869 (+ 46789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1712317}
I20250112 20:51:35.071153 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 397) took 1606 ms. Trace:
I20250112 20:51:35.071257 1186 rpcz_store.cc:276] 0112 20:51:33.464802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.464863 (+ 61us) service_pool.cc:225] Handling call
0112 20:51:35.070972 (+1606109us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:35.070981 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.071081 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597040}
I20250112 20:51:35.071589 1186 tablet_replica.cc:331] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.071339 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e72fda1743fc4f96b29add6e36e884de (table default.Table_15828ef1bbbc41068bb6b6f2fbb18f94 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:35.071805 1186 raft_consensus.cc:2238] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.072024 1186 raft_consensus.cc:2267] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.072230 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4d29316b9d2441bca97f56bdb0e56f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d29316b9d2441bca97f56bdb0e56f0d already in progress: creating tablet
I20250112 20:51:35.072352 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.072605 1186 ts_tablet_manager.cc:1905] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.072759 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.072736 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 65784493c6ef417993b1f6371cab47d6 (DEFAULT_TABLE table=default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 [id=4130b8a93c4645839e4fad41a2127529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.073045 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 615d75bc22df45cc876788a266e983cc (DEFAULT_TABLE table=default.Table_b5d6bc8b522a4be197bd045c77d4a5f4 [id=3066f1ca47fe4a39841cfd5b6028f16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.073180 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65784493c6ef417993b1f6371cab47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.073402 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615d75bc22df45cc876788a266e983cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.073675 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a20d85a4be874b0b90c528a631a29aed (DEFAULT_TABLE table=default.Table_ec3a27d13db349b4a78c058fdea0ad2d [id=c628277a9fdf4ff7b2df5f322a8f5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.073980 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20d85a4be874b0b90c528a631a29aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.087095 1163 tablet_bootstrap.cc:492] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.087270 1163 ts_tablet_manager.cc:1397] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.002s
I20250112 20:51:35.088181 1163 raft_consensus.cc:357] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.088398 1163 raft_consensus.cc:383] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.088483 1163 raft_consensus.cc:738] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.092247 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 07c83f45bd3d469ca55b3d9a5810d27d (DEFAULT_TABLE table=default.Table_cdede98843a14d8e83eb0d3a74827353 [id=e5b078f9a27a45e69769ad7e9f5c1ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.092615 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c83f45bd3d469ca55b3d9a5810d27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.096698 884 tablet_service.cc:1467] Processing CreateTablet for tablet 07c83f45bd3d469ca55b3d9a5810d27d (DEFAULT_TABLE table=default.Table_cdede98843a14d8e83eb0d3a74827353 [id=e5b078f9a27a45e69769ad7e9f5c1ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.097478 874 tablet_service.cc:1467] Processing CreateTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.098193 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 5d167065029a46b791d413db8a2c4bcc (DEFAULT_TABLE table=default.Table_6e73aeaec7af4cd0952e8f427103f61f [id=0f5678b6533e42d190d24c6958091caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.098203 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5d167065029a46b791d413db8a2c4bcc (DEFAULT_TABLE table=default.Table_6e73aeaec7af4cd0952e8f427103f61f [id=0f5678b6533e42d190d24c6958091caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.098594 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d167065029a46b791d413db8a2c4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.099872 1017 tablet_service.cc:1467] Processing CreateTablet for tablet db4fa71cf46144629e1fc1f2172fc3cb (DEFAULT_TABLE table=default.Table_5dbbb84fc94a4e88bb3331103463e4da [id=60b7060bed91463c89327b040c5c4513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.100265 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db4fa71cf46144629e1fc1f2172fc3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.102182 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 390) took 1474 ms. Trace:
I20250112 20:51:35.102283 1184 rpcz_store.cc:276] 0112 20:51:33.627517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.628184 (+ 667us) service_pool.cc:225] Handling call
0112 20:51:34.954907 (+1326723us) ts_tablet_manager.cc:1145] Deleting tablet a48df2ec556d43a3b22129b1177ebf74
0112 20:51:34.954918 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:34.976484 (+ 21566us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:34.982903 (+ 6419us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.102123 (+119220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326504}
I20250112 20:51:35.102699 877 tablet_service.cc:1467] Processing CreateTablet for tablet db4fa71cf46144629e1fc1f2172fc3cb (DEFAULT_TABLE table=default.Table_5dbbb84fc94a4e88bb3331103463e4da [id=60b7060bed91463c89327b040c5c4513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.088742 1163 consensus_queue.cc:260] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.104413 1024 tablet_service.cc:1467] Processing CreateTablet for tablet bf66492d01a64eb1b842acc279b8ee09 (DEFAULT_TABLE table=default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a [id=e1d57fdfdc434d36bb69c54d5c974f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.104449 882 tablet_service.cc:1467] Processing CreateTablet for tablet bf66492d01a64eb1b842acc279b8ee09 (DEFAULT_TABLE table=default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a [id=e1d57fdfdc434d36bb69c54d5c974f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.104830 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66492d01a64eb1b842acc279b8ee09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.105146 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 391) took 1377 ms. Trace:
I20250112 20:51:35.105654 893 tablet_service.cc:1467] Processing CreateTablet for tablet db9bab2477a94c169817d9de8aaad1c0 (DEFAULT_TABLE table=default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.106127 880 tablet_service.cc:1467] Processing CreateTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 (DEFAULT_TABLE table=default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.106539 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d167065029a46b791d413db8a2c4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.106541 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet a48df2ec556d43a3b22129b1177ebf74 (table default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]) successfully deleted
I20250112 20:51:35.105247 1184 rpcz_store.cc:276] 0112 20:51:33.727721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.727810 (+ 89us) service_pool.cc:225] Handling call
0112 20:51:35.104976 (+1377166us) ts_tablet_manager.cc:1145] Deleting tablet 9c6d5d425e6a493daa46900cbd8db166
0112 20:51:35.104985 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.105092 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340914}
I20250112 20:51:35.107367 888 tablet_service.cc:1467] Processing CreateTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 (DEFAULT_TABLE table=default.Table_be8270d1ca354e748dac253a4aeb09a3 [id=a63f77f9d86346d0aed333ccca1a28ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.107601 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 392) took 1361 ms. Trace:
I20250112 20:51:35.107676 1163 ts_tablet_manager.cc:1428] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20250112 20:51:35.108083 1163 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.108278 1028 tablet_service.cc:1467] Processing CreateTablet for tablet db9bab2477a94c169817d9de8aaad1c0 (DEFAULT_TABLE table=default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.108594 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9bab2477a94c169817d9de8aaad1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.109412 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 (DEFAULT_TABLE table=default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.109777 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9be1f065ec4605a0a3fe69770793d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.113174 1178 tablet_bootstrap.cc:654] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.114394 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 (DEFAULT_TABLE table=default.Table_be8270d1ca354e748dac253a4aeb09a3 [id=a63f77f9d86346d0aed333ccca1a28ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.114802 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725e56fa8ac343e5b1ec1214da02b7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.057075 741 tablet_service.cc:1467] Processing CreateTablet for tablet b37b00c49f234dc89d63f644febc23fb (DEFAULT_TABLE table=default.Table_e907fd649a434fe9b3a51b2619964cb3 [id=8997a20696974e4fb75b4f18903d7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.116375 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37b00c49f234dc89d63f644febc23fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.116397 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9be1f065ec4605a0a3fe69770793d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.116608 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9bab2477a94c169817d9de8aaad1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.116915 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db4fa71cf46144629e1fc1f2172fc3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.057446 745 tablet_service.cc:1467] Processing CreateTablet for tablet a20d85a4be874b0b90c528a631a29aed (DEFAULT_TABLE table=default.Table_ec3a27d13db349b4a78c058fdea0ad2d [id=c628277a9fdf4ff7b2df5f322a8f5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.121109 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20d85a4be874b0b90c528a631a29aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.121585 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725e56fa8ac343e5b1ec1214da02b7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.121798 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.122709 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66492d01a64eb1b842acc279b8ee09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.123440 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c83f45bd3d469ca55b3d9a5810d27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.057755 752 tablet_service.cc:1467] Processing CreateTablet for tablet 65784493c6ef417993b1f6371cab47d6 (DEFAULT_TABLE table=default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 [id=4130b8a93c4645839e4fad41a2127529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.124951 746 tablet_service.cc:1467] Processing CreateTablet for tablet d738720dd7414764b4d1191cdc37895d (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.125054 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65784493c6ef417993b1f6371cab47d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.125301 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d738720dd7414764b4d1191cdc37895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.065500 754 tablet_service.cc:1467] Processing CreateTablet for tablet 615d75bc22df45cc876788a266e983cc (DEFAULT_TABLE table=default.Table_b5d6bc8b522a4be197bd045c77d4a5f4 [id=3066f1ca47fe4a39841cfd5b6028f16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.125766 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615d75bc22df45cc876788a266e983cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.066386 737 tablet_service.cc:1467] Processing CreateTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.077643 1164 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.129012 1164 ts_tablet_manager.cc:1397] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.000s
I20250112 20:51:35.129730 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.130482 1164 raft_consensus.cc:357] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.130707 1164 raft_consensus.cc:383] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.130806 1164 raft_consensus.cc:738] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.131071 1164 consensus_queue.cc:260] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.131628 1164 ts_tablet_manager.cc:1428] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:35.131961 1164 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.089851 753 tablet_service.cc:1467] Processing CreateTablet for tablet 07c83f45bd3d469ca55b3d9a5810d27d (DEFAULT_TABLE table=default.Table_cdede98843a14d8e83eb0d3a74827353 [id=e5b078f9a27a45e69769ad7e9f5c1ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.096254 738 tablet_service.cc:1467] Processing CreateTablet for tablet 5d167065029a46b791d413db8a2c4bcc (DEFAULT_TABLE table=default.Table_6e73aeaec7af4cd0952e8f427103f61f [id=0f5678b6533e42d190d24c6958091caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.099062 756 tablet_service.cc:1467] Processing CreateTablet for tablet db4fa71cf46144629e1fc1f2172fc3cb (DEFAULT_TABLE table=default.Table_5dbbb84fc94a4e88bb3331103463e4da [id=60b7060bed91463c89327b040c5c4513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.108747 749 tablet_service.cc:1467] Processing CreateTablet for tablet bf66492d01a64eb1b842acc279b8ee09 (DEFAULT_TABLE table=default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a [id=e1d57fdfdc434d36bb69c54d5c974f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.109050 744 tablet_service.cc:1467] Processing CreateTablet for tablet db9bab2477a94c169817d9de8aaad1c0 (DEFAULT_TABLE table=default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.109452 747 tablet_service.cc:1467] Processing CreateTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 (DEFAULT_TABLE table=default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.109776 739 tablet_service.cc:1467] Processing CreateTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 (DEFAULT_TABLE table=default.Table_be8270d1ca354e748dac253a4aeb09a3 [id=a63f77f9d86346d0aed333ccca1a28ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.133728 1164 tablet_bootstrap.cc:654] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.139596 1186 ts_tablet_manager.cc:1918] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.139739 1186 log.cc:1198] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/a48df2ec556d43a3b22129b1177ebf74
I20250112 20:51:35.140185 1186 ts_tablet_manager.cc:1939] T a48df2ec556d43a3b22129b1177ebf74 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.141103 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 398) took 1512 ms. Trace:
I20250112 20:51:35.142494 1186 rpcz_store.cc:276] 0112 20:51:33.628743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.628814 (+ 71us) service_pool.cc:225] Handling call
0112 20:51:35.071511 (+1442697us) ts_tablet_manager.cc:1145] Deleting tablet a48df2ec556d43a3b22129b1177ebf74
0112 20:51:35.071520 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.079767 (+ 8247us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.139436 (+ 59669us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.141066 (+ 1630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442484}
I20250112 20:51:35.143116 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 399) took 1414 ms. Trace:
I20250112 20:51:35.143253 1186 rpcz_store.cc:276] 0112 20:51:33.728555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.728604 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:35.142944 (+1414340us) ts_tablet_manager.cc:1145] Deleting tablet 9c6d5d425e6a493daa46900cbd8db166
0112 20:51:35.142958 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.143079 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414088}
I20250112 20:51:35.143775 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 400) took 1398 ms. Trace:
I20250112 20:51:35.143888 1186 rpcz_store.cc:276] 0112 20:51:33.745426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.745513 (+ 87us) service_pool.cc:225] Handling call
0112 20:51:35.143642 (+1398129us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:35.143651 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.143731 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397924}
I20250112 20:51:35.144400 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 402) took 1298 ms. Trace:
I20250112 20:51:35.144510 1186 rpcz_store.cc:276] 0112 20:51:33.845712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.845985 (+ 273us) service_pool.cc:225] Handling call
0112 20:51:35.144267 (+1298282us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:35.144275 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.144355 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222994}
I20250112 20:51:35.146090 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d738720dd7414764b4d1191cdc37895d (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.146508 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d738720dd7414764b4d1191cdc37895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.144965 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 404) took 1284 ms. Trace:
I20250112 20:51:35.146687 1186 rpcz_store.cc:276] 0112 20:51:33.860169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.860226 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:35.144852 (+1284626us) ts_tablet_manager.cc:1145] Deleting tablet 2469127891524be5ac3dc13fbcf35c7a
0112 20:51:35.144860 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.144937 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223382}
I20250112 20:51:35.147064 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 403) took 1288 ms. Trace:
I20250112 20:51:35.147183 1186 rpcz_store.cc:276] 0112 20:51:33.858579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.860918 (+ 2339us) service_pool.cc:225] Handling call
0112 20:51:35.146936 (+1286018us) ts_tablet_manager.cc:1145] Deleting tablet 4561956c97824824a55283a8da21929e
0112 20:51:35.146947 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.147036 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225338}
I20250112 20:51:35.147557 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 405) took 1282 ms. Trace:
I20250112 20:51:35.147665 1186 rpcz_store.cc:276] 0112 20:51:33.865005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.865239 (+ 234us) service_pool.cc:225] Handling call
0112 20:51:35.147408 (+1282169us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:35.147416 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.147524 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225749}
I20250112 20:51:35.148581 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c83f45bd3d469ca55b3d9a5810d27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.146519 1821 raft_consensus.cc:491] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.148976 1821 raft_consensus.cc:513] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.149612 1821 leader_election.cc:290] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.149737 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db4fa71cf46144629e1fc1f2172fc3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.150298 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:35.150878 1186 tablet_replica.cc:331] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.151160 1186 raft_consensus.cc:2238] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.151345 1186 raft_consensus.cc:2267] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.152199 1186 ts_tablet_manager.cc:1905] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.158136 1186 ts_tablet_manager.cc:1918] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.158279 1186 log.cc:1198] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/201a7e2f17bb46fa918dad8f878a4065
I20250112 20:51:35.158737 1186 ts_tablet_manager.cc:1939] T 201a7e2f17bb46fa918dad8f878a4065 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.161198 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet a48df2ec556d43a3b22129b1177ebf74 (table default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]) successfully deleted
W20250112 20:51:35.161494 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9c6d5d425e6a493daa46900cbd8db166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c6d5d425e6a493daa46900cbd8db166 already in progress: creating tablet
W20250112 20:51:35.161780 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e9f669a739d94f99a5b7e9ccccd29545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9f669a739d94f99a5b7e9ccccd29545 already in progress: creating tablet
W20250112 20:51:35.162164 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2ab635fec8a64efd8e0c527d913ccd7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ab635fec8a64efd8e0c527d913ccd7d already in progress: creating tablet
W20250112 20:51:35.162532 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2469127891524be5ac3dc13fbcf35c7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2469127891524be5ac3dc13fbcf35c7a already in progress: creating tablet
W20250112 20:51:35.162779 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4561956c97824824a55283a8da21929e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4561956c97824824a55283a8da21929e already in progress: creating tablet
W20250112 20:51:35.163050 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4f34aa32d6474561988b2cf3d0e8fd3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f34aa32d6474561988b2cf3d0e8fd3b already in progress: creating tablet
W20250112 20:51:35.163280 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1fd47d1c1a4b42bdbff97f619dc95134 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd47d1c1a4b42bdbff97f619dc95134 already in progress: creating tablet
I20250112 20:51:35.163712 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9be1f065ec4605a0a3fe69770793d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.173164 885 tablet_service.cc:1467] Processing CreateTablet for tablet d738720dd7414764b4d1191cdc37895d (DEFAULT_TABLE table=default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.174193 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725e56fa8ac343e5b1ec1214da02b7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.174315 1932 raft_consensus.cc:491] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.174503 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d167065029a46b791d413db8a2c4bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.174490 1932 raft_consensus.cc:513] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.175422 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db9bab2477a94c169817d9de8aaad1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.175781 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d738720dd7414764b4d1191cdc37895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.176398 1163 tablet_bootstrap.cc:654] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.179994 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.180251 1046 raft_consensus.cc:2463] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:35.107697 1184 rpcz_store.cc:276] 0112 20:51:33.745827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.746463 (+ 636us) service_pool.cc:225] Handling call
0112 20:51:35.107456 (+1360993us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:35.107464 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.107545 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340466}
I20250112 20:51:35.181804 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 394) took 1336 ms. Trace:
I20250112 20:51:35.181938 1184 rpcz_store.cc:276] 0112 20:51:33.845718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.845774 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:35.181614 (+1335840us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:35.181623 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.181754 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335656}
I20250112 20:51:35.182519 1184 tablet_replica.cc:331] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.182735 1184 raft_consensus.cc:2238] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.182960 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf66492d01a64eb1b842acc279b8ee09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.183109 1184 raft_consensus.cc:2267] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.181110 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:35.186630 1186 tablet_replica.cc:331] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.186936 1186 raft_consensus.cc:2238] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.187251 1186 raft_consensus.cc:2267] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.188339 1186 ts_tablet_manager.cc:1905] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.188930 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a7e2f17bb46fa918dad8f878a4065" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.195170 1186 ts_tablet_manager.cc:1918] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.195302 1186 log.cc:1198] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/70c86ad5f59d431e9b1a1fa890fd35c0
I20250112 20:51:35.197484 1186 ts_tablet_manager.cc:1939] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.198873 1186 tablet_replica.cc:331] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.201066 1186 raft_consensus.cc:2238] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.199119 1178 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.201599 1178 ts_tablet_manager.cc:1397] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.193s user 0.002s sys 0.002s
I20250112 20:51:35.202591 1186 raft_consensus.cc:2267] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.203145 1186 ts_tablet_manager.cc:1905] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.202363 1178 raft_consensus.cc:357] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.204224 1178 raft_consensus.cc:383] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.204322 1178 raft_consensus.cc:738] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.204577 1178 consensus_queue.cc:260] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.207300 1184 ts_tablet_manager.cc:1905] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.209264 1178 ts_tablet_manager.cc:1428] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250112 20:51:35.212965 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 49672.7us
W20250112 20:51:35.213295 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1fd34e829d0645ea80daaff5d27cb266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd34e829d0645ea80daaff5d27cb266 already in progress: creating tablet
I20250112 20:51:35.215507 1932 leader_election.cc:290] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.215673 710 leader_election.cc:304] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:35.216043 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 already in progress: creating tablet
W20250112 20:51:35.216266 710 leader_election.cc:343] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:35.216246 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 49b8cc99742f4f9ba68eb1aa17cab248 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b8cc99742f4f9ba68eb1aa17cab248 already in progress: creating tablet
I20250112 20:51:35.216504 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 201a7e2f17bb46fa918dad8f878a4065 (table default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]) successfully deleted
W20250112 20:51:35.216691 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 669e1cfd407e4d87a63e6832d9c6fa1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 669e1cfd407e4d87a63e6832d9c6fa1f already in progress: creating tablet
I20250112 20:51:35.216665 983 leader_election.cc:304] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:35.217442 1821 raft_consensus.cc:2743] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.217551 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (table default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]) successfully deleted
W20250112 20:51:35.217743 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f0daaeb0bf6046da8de969b1f4c71c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0daaeb0bf6046da8de969b1f4c71c84 already in progress: creating tablet
W20250112 20:51:35.221053 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9c6d5d425e6a493daa46900cbd8db166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c6d5d425e6a493daa46900cbd8db166 already in progress: creating tablet
I20250112 20:51:35.221570 1932 raft_consensus.cc:2798] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:35.221680 1932 raft_consensus.cc:491] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:35.221751 1932 raft_consensus.cc:3054] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:35.221905 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e9f669a739d94f99a5b7e9ccccd29545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9f669a739d94f99a5b7e9ccccd29545 already in progress: creating tablet
W20250112 20:51:35.222124 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2ab635fec8a64efd8e0c527d913ccd7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ab635fec8a64efd8e0c527d913ccd7d already in progress: creating tablet
I20250112 20:51:35.223189 1184 ts_tablet_manager.cc:1918] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.223295 1184 log.cc:1198] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4c3f359655c94f1d905943069548845b
I20250112 20:51:35.223855 1184 ts_tablet_manager.cc:1939] T 4c3f359655c94f1d905943069548845b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.224573 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 395) took 1374 ms. Trace:
I20250112 20:51:35.224762 1184 rpcz_store.cc:276] 0112 20:51:33.850569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.850624 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:35.182357 (+1331733us) ts_tablet_manager.cc:1145] Deleting tablet 4c3f359655c94f1d905943069548845b
0112 20:51:35.182365 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.210394 (+ 28029us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.223069 (+ 12675us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.224535 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331517}
I20250112 20:51:35.225044 1932 raft_consensus.cc:513] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.225553 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 397) took 1365 ms. Trace:
I20250112 20:51:35.225854 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4c3f359655c94f1d905943069548845b (table default.Table_ab141f89a5bc48c1b0e8d83f0975d74f [id=1a369ac3a4124715977098224508590c]) successfully deleted
I20250112 20:51:35.226300 1184 rpcz_store.cc:276] 0112 20:51:33.859688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.861285 (+ 1597us) service_pool.cc:225] Handling call
0112 20:51:35.225395 (+1364110us) ts_tablet_manager.cc:1145] Deleting tablet 2469127891524be5ac3dc13fbcf35c7a
0112 20:51:35.225404 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.225510 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363873}
W20250112 20:51:35.227031 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2469127891524be5ac3dc13fbcf35c7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2469127891524be5ac3dc13fbcf35c7a already in progress: creating tablet
I20250112 20:51:35.228029 747 tablet_service.cc:1467] Processing CreateTablet for tablet 691e2c3f72ab4f439678d166b9b31661 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.228473 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691e2c3f72ab4f439678d166b9b31661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.229043 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:35.229276 1048 raft_consensus.cc:3054] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.229167 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 396) took 1369 ms. Trace:
I20250112 20:51:35.230091 1932 leader_election.cc:290] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.229950 1184 rpcz_store.cc:276] 0112 20:51:33.859540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.863880 (+ 4340us) service_pool.cc:225] Handling call
0112 20:51:35.228169 (+1364289us) ts_tablet_manager.cc:1145] Deleting tablet 4561956c97824824a55283a8da21929e
0112 20:51:35.228255 (+ 86us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.229023 (+ 768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363991}
W20250112 20:51:35.231386 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4561956c97824824a55283a8da21929e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4561956c97824824a55283a8da21929e already in progress: creating tablet
I20250112 20:51:35.232612 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 398) took 1367 ms. Trace:
I20250112 20:51:35.233062 738 tablet_service.cc:1467] Processing CreateTablet for tablet 263b370f96044305a79ebdb79d5e01d1 (DEFAULT_TABLE table=default.Table_2ce57ea2a0d44a2d817812e80fe5d573 [id=e3102937d2784e4d95172093df1fa2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.233258 884 tablet_service.cc:1467] Processing CreateTablet for tablet 691e2c3f72ab4f439678d166b9b31661 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.233664 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263b370f96044305a79ebdb79d5e01d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.233507 1184 rpcz_store.cc:276] 0112 20:51:33.864757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.867323 (+ 2566us) service_pool.cc:225] Handling call
0112 20:51:35.232013 (+1364690us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:35.232063 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.232475 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364392}
I20250112 20:51:35.235078 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691e2c3f72ab4f439678d166b9b31661. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:35.235457 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4f34aa32d6474561988b2cf3d0e8fd3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f34aa32d6474561988b2cf3d0e8fd3b already in progress: creating tablet
I20250112 20:51:35.237226 1185 ts_tablet_manager.cc:1939] T 5ca061b82a894e91b9508613ccd8ecd7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:35.237262 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1fd47d1c1a4b42bdbff97f619dc95134 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd47d1c1a4b42bdbff97f619dc95134 already in progress: creating tablet
I20250112 20:51:35.238199 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 384) took 2081 ms. Trace:
I20250112 20:51:35.238365 1185 rpcz_store.cc:276] 0112 20:51:33.157007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.158499 (+ 1492us) service_pool.cc:225] Handling call
0112 20:51:35.029192 (+1870693us) ts_tablet_manager.cc:1145] Deleting tablet 5ca061b82a894e91b9508613ccd8ecd7
0112 20:51:35.029200 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.053391 (+ 24191us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.056426 (+ 3035us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.238159 (+181733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862456}
W20250112 20:51:35.238718 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 already in progress: creating tablet
I20250112 20:51:35.239045 1185 tablet_replica.cc:331] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.239097 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5ca061b82a894e91b9508613ccd8ecd7 (table default.Table_c275eb298fde46e7b70911f21c75fffe [id=e21586dd18394721b1e0e0a5df69259a]) successfully deleted
I20250112 20:51:35.239367 1185 raft_consensus.cc:2238] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.239604 1185 raft_consensus.cc:2267] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.240419 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1fd34e829d0645ea80daaff5d27cb266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd34e829d0645ea80daaff5d27cb266 already in progress: creating tablet
I20250112 20:51:35.240600 1185 ts_tablet_manager.cc:1905] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.240922 1178 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.241930 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8330fa1e4654a1a9f167a1120a103fc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:35.242767 710 leader_election.cc:343] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.242815 1184 tablet_replica.cc:331] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:35.243506 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 49b8cc99742f4f9ba68eb1aa17cab248 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b8cc99742f4f9ba68eb1aa17cab248 already in progress: creating tablet
I20250112 20:51:35.243851 1178 tablet_bootstrap.cc:654] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.244558 1982 raft_consensus.cc:491] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.244166 1184 raft_consensus.cc:2238] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.245285 1982 raft_consensus.cc:513] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.245488 1186 ts_tablet_manager.cc:1918] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:35.232795 1048 raft_consensus.cc:2463] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:35.246222 1186 log.cc:1198] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/decdec0d34344ec4b422643b0d4e7659
I20250112 20:51:35.248157 1163 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.248173 710 leader_election.cc:304] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.248376 1163 ts_tablet_manager.cc:1397] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.000s
I20250112 20:51:35.248878 1931 raft_consensus.cc:2798] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:35.249034 1931 raft_consensus.cc:695] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:35.249089 874 tablet_service.cc:1467] Processing CreateTablet for tablet 263b370f96044305a79ebdb79d5e01d1 (DEFAULT_TABLE table=default.Table_2ce57ea2a0d44a2d817812e80fe5d573 [id=e3102937d2784e4d95172093df1fa2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.249411 1186 ts_tablet_manager.cc:1939] T decdec0d34344ec4b422643b0d4e7659 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.249481 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263b370f96044305a79ebdb79d5e01d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.249341 1931 consensus_queue.cc:237] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.252373 1185 ts_tablet_manager.cc:1918] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.252488 1185 log.cc:1198] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:35.252895 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4688dd6bbfc046b7bb140206cae9f104" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.253461 1186 tablet_replica.cc:331] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.254207 1186 raft_consensus.cc:2238] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.252306 1982 leader_election.cc:290] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.254571 1186 pending_rounds.cc:62] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.254637 1186 pending_rounds.cc:69] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585028713132032 op_type: NO_OP noop_request { }
I20250112 20:51:35.254711 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4688dd6bbfc046b7bb140206cae9f104" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.255043 1186 raft_consensus.cc:2883] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.255172 1186 raft_consensus.cc:2267] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.255487 847 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:35.255965 847 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.256151 1185 ts_tablet_manager.cc:1939] T e72fda1743fc4f96b29add6e36e884de P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.256129 847 leader_election.cc:304] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:35.256419 1982 raft_consensus.cc:2743] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.261528 1186 ts_tablet_manager.cc:1905] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.266126 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 263b370f96044305a79ebdb79d5e01d1 (DEFAULT_TABLE table=default.Table_2ce57ea2a0d44a2d817812e80fe5d573 [id=e3102937d2784e4d95172093df1fa2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.257282 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 386) took 2099 ms. Trace:
I20250112 20:51:35.266599 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 263b370f96044305a79ebdb79d5e01d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.266497 1185 rpcz_store.cc:276] 0112 20:51:33.157297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.157481 (+ 184us) service_pool.cc:225] Handling call
0112 20:51:35.238942 (+2081461us) ts_tablet_manager.cc:1145] Deleting tablet e72fda1743fc4f96b29add6e36e884de
0112 20:51:35.238999 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.244506 (+ 5507us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.252272 (+ 7766us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.257205 (+ 4933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066735}
I20250112 20:51:35.267669 1185 tablet_replica.cc:331] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.267999 1185 raft_consensus.cc:2238] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.268287 1185 raft_consensus.cc:2267] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.269026 1185 ts_tablet_manager.cc:1905] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.272228 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 691e2c3f72ab4f439678d166b9b31661 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.272738 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691e2c3f72ab4f439678d166b9b31661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.275840 1185 ts_tablet_manager.cc:1918] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 20:51:35.277740 1186 ts_tablet_manager.cc:1918] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.277844 1185 log.cc:1198] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5f012db68caf4d7d86afa6deb9f2ffaa
I20250112 20:51:35.278224 1186 log.cc:1198] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:35.278571 1163 raft_consensus.cc:357] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.278805 1186 ts_tablet_manager.cc:1939] T e590f9615c4d47abaad1a36ac74b3c80 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.278818 1163 raft_consensus.cc:383] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.279063 1163 raft_consensus.cc:738] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.279127 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_09c966930c7b4277a84b783534270688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:35.279259 1163 consensus_queue.cc:260] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.277119 1164 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:35.279760 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09c966930c7b4277a84b783534270688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.279793 1164 ts_tablet_manager.cc:1397] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.003s
I20250112 20:51:35.279861 1163 ts_tablet_manager.cc:1428] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250112 20:51:35.280186 1163 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.280623 1164 raft_consensus.cc:357] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.280861 1164 raft_consensus.cc:383] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.280953 1164 raft_consensus.cc:738] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.281574 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_66ed5acb10ac4640a299d0b36ebc314b" } modify_external_catalogs: true
I20250112 20:51:35.281752 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_66ed5acb10ac4640a299d0b36ebc314b" } modify_external_catalogs: true
I20250112 20:51:35.282058 1185 ts_tablet_manager.cc:1939] T 5f012db68caf4d7d86afa6deb9f2ffaa P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.282188 1163 tablet_bootstrap.cc:654] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.282246 1186 tablet_replica.cc:331] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.282430 1186 raft_consensus.cc:2238] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.282604 1186 raft_consensus.cc:2267] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.282820 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 387) took 2050 ms. Trace:
I20250112 20:51:35.283788 1185 rpcz_store.cc:276] 0112 20:51:33.232259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.232316 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:35.267094 (+2034778us) ts_tablet_manager.cc:1145] Deleting tablet 5f012db68caf4d7d86afa6deb9f2ffaa
0112 20:51:35.267105 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.272489 (+ 5384us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.275707 (+ 3218us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.282785 (+ 7078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012540}
W20250112 20:51:35.284129 983 consensus_peers.cc:487] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:35.278759 1184 raft_consensus.cc:2267] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.287536 1163 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.287731 1163 ts_tablet_manager.cc:1397] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250112 20:51:35.281186 1164 consensus_queue.cc:260] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.288410 1163 raft_consensus.cc:357] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.288558 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 389) took 1838 ms. Trace:
I20250112 20:51:35.288619 1163 raft_consensus.cc:383] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.288707 1163 raft_consensus.cc:738] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.289358 1163 consensus_queue.cc:260] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.289803 1164 ts_tablet_manager.cc:1428] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:35.290076 1163 ts_tablet_manager.cc:1428] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:35.290402 1163 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.289239 1185 rpcz_store.cc:276] 0112 20:51:33.450588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.450651 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:35.288381 (+1837730us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:35.288391 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.288526 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1837475}
I20250112 20:51:35.291484 1164 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.292570 1163 tablet_bootstrap.cc:654] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.293440 1164 tablet_bootstrap.cc:654] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.291781 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 390) took 1705 ms. Trace:
I20250112 20:51:35.294413 1185 rpcz_store.cc:276] 0112 20:51:33.585946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.587653 (+ 1707us) service_pool.cc:225] Handling call
0112 20:51:35.291647 (+1703994us) ts_tablet_manager.cc:1145] Deleting tablet bcbbefa7b55041a6a6b50b3c96c45833
0112 20:51:35.291656 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.291750 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1703529}
I20250112 20:51:35.294932 1185 tablet_replica.cc:331] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.295775 1185 raft_consensus.cc:2238] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.296025 1185 raft_consensus.cc:2267] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.296648 1185 ts_tablet_manager.cc:1905] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.298614 1186 ts_tablet_manager.cc:1905] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.302016 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1d97ac93467a4a7a8ae44d52719f191c
I20250112 20:51:35.302453 447 catalog_manager.cc:6596] ReplaceTablet: tablet 1d97ac93467a4a7a8ae44d52719f191c deleted and replaced by tablet f52a9216c5044f88a0060e3e8e95eddd
I20250112 20:51:35.304769 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1d60604713524dba8f3a42c4f3bc7300
I20250112 20:51:35.297483 1184 ts_tablet_manager.cc:1905] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.305464 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_10962594cedf43f6a5533239b618afa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.305967 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10962594cedf43f6a5533239b618afa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.306552 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e72fda1743fc4f96b29add6e36e884de (table default.Table_15828ef1bbbc41068bb6b6f2fbb18f94 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:35.307202 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5f012db68caf4d7d86afa6deb9f2ffaa (table default.Table_a77dee800e62415db3cada3902c1efcc [id=9448c50d601f435e81de18e95f86efb0]) successfully deleted
I20250112 20:51:35.307765 1185 ts_tablet_manager.cc:1918] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.309419 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9570269368e14b1abeaffdcc741c6f3a
I20250112 20:51:35.311151 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d17e6d263014448b96c5d1f03013e6e7
I20250112 20:51:35.312296 1164 tablet_bootstrap.cc:492] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.312484 1164 ts_tablet_manager.cc:1397] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250112 20:51:35.312666 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d7117edf078848c0a1c83ab8e804dfed
I20250112 20:51:35.313243 448 catalog_manager.cc:6596] ReplaceTablet: tablet d7117edf078848c0a1c83ab8e804dfed deleted and replaced by tablet e2c41c2c92b442b986d6b75708c7420c
I20250112 20:51:35.314320 1164 raft_consensus.cc:357] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.314612 1164 raft_consensus.cc:383] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.314688 1164 raft_consensus.cc:738] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.314991 1164 consensus_queue.cc:260] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.316241 1164 ts_tablet_manager.cc:1428] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250112 20:51:35.316903 1164 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.317350 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 321b74c1358040d7ad3a06cca6365e73
I20250112 20:51:35.318431 1185 log.cc:1198] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/a48df2ec556d43a3b22129b1177ebf74
I20250112 20:51:35.318903 1185 ts_tablet_manager.cc:1939] T a48df2ec556d43a3b22129b1177ebf74 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.319680 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 393) took 1668 ms. Trace:
I20250112 20:51:35.318295 447 catalog_manager.cc:5526] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:35.319804 1185 rpcz_store.cc:276] 0112 20:51:33.651551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.652036 (+ 485us) service_pool.cc:225] Handling call
0112 20:51:35.294864 (+1642828us) ts_tablet_manager.cc:1145] Deleting tablet a48df2ec556d43a3b22129b1177ebf74
0112 20:51:35.294872 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.303097 (+ 8225us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.307625 (+ 4528us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.319629 (+ 12004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642012}
I20250112 20:51:35.320489 1186 ts_tablet_manager.cc:1918] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.320647 1186 log.cc:1198] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:35.320712 1164 tablet_bootstrap.cc:654] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.322583 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8a830df519164b7b89e43fa9335a1a20" } new_table_name: "default.Table_0466e8f4f9b14dc8b813c109373e1d1e" modify_external_catalogs: true
I20250112 20:51:35.323832 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_14ab0d3d40e14fdfaec2dfe76fd50455" } new_table_name: "default.Table_8f4ca7a462ed41ff8df87e593847cd3a" modify_external_catalogs: true
I20250112 20:51:35.323834 1982 consensus_queue.cc:1035] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.324543 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f59b364b3b3490aaa782d7e83f048ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.325084 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f59b364b3b3490aaa782d7e83f048ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.325800 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a28d1068edaf4014a962ab336f9243d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:35.326314 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_71108c2ed35249678274682b502f927a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.326646 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a28d1068edaf4014a962ab336f9243d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:35.326709 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71108c2ed35249678274682b502f927a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.327845 1186 ts_tablet_manager.cc:1939] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.328154 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_23882b539e614fcca89f4d9b2fadd948" } new_table_name: "default.Table_292dce92952a4a3ba27a7bb1ed129e3e" modify_external_catalogs: true
I20250112 20:51:35.330024 1186 tablet_replica.cc:331] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.329828 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 394) took 1602 ms. Trace:
I20250112 20:51:35.330423 1185 rpcz_store.cc:276] 0112 20:51:33.727263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.727315 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:35.329612 (+1602297us) ts_tablet_manager.cc:1145] Deleting tablet 9c6d5d425e6a493daa46900cbd8db166
0112 20:51:35.329627 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.329748 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602097}
I20250112 20:51:35.330711 1186 raft_consensus.cc:2238] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.331012 1186 pending_rounds.cc:62] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.332285 1932 consensus_queue.cc:1035] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.333014 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 395) took 1588 ms. Trace:
I20250112 20:51:35.333160 1185 rpcz_store.cc:276] 0112 20:51:33.744509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.744640 (+ 131us) service_pool.cc:225] Handling call
0112 20:51:35.332863 (+1588223us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:35.332873 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.332965 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587964}
W20250112 20:51:35.333123 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4d29316b9d2441bca97f56bdb0e56f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d29316b9d2441bca97f56bdb0e56f0d already in progress: creating tablet
I20250112 20:51:35.333709 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 397) took 1487 ms. Trace:
I20250112 20:51:35.333863 1185 rpcz_store.cc:276] 0112 20:51:33.845913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.846220 (+ 307us) service_pool.cc:225] Handling call
0112 20:51:35.333561 (+1487341us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:35.333570 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.333660 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1487136}
I20250112 20:51:35.334390 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 401) took 1469 ms. Trace:
I20250112 20:51:35.334524 1185 rpcz_store.cc:276] 0112 20:51:33.865181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.865420 (+ 239us) service_pool.cc:225] Handling call
0112 20:51:35.334241 (+1468821us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:35.334250 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.334341 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468613}
I20250112 20:51:35.334826 1186 pending_rounds.cc:69] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585026465624064 op_type: NO_OP noop_request { }
I20250112 20:51:35.335304 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 399) took 1476 ms. Trace:
I20250112 20:51:35.335474 1186 raft_consensus.cc:2883] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.335451 1185 rpcz_store.cc:276] 0112 20:51:33.859085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.859136 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:35.335172 (+1476036us) ts_tablet_manager.cc:1145] Deleting tablet 4561956c97824824a55283a8da21929e
0112 20:51:35.335179 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.335256 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462810}
I20250112 20:51:35.335999 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 400) took 1476 ms. Trace:
I20250112 20:51:35.336015 1186 raft_consensus.cc:2267] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.335860 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bcbbefa7b55041a6a6b50b3c96c45833 because the tablet was not found. No further retry: Not found: Tablet not found: bcbbefa7b55041a6a6b50b3c96c45833
I20250112 20:51:35.337123 1185 rpcz_store.cc:276] 0112 20:51:33.859832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:33.859883 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:35.335832 (+1475949us) ts_tablet_manager.cc:1145] Deleting tablet 2469127891524be5ac3dc13fbcf35c7a
0112 20:51:35.335859 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.335936 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463416}
I20250112 20:51:35.337057 1164 tablet_bootstrap.cc:492] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.337543 1186 ts_tablet_manager.cc:1905] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.337620 1164 ts_tablet_manager.cc:1397] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20250112 20:51:35.337874 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet a48df2ec556d43a3b22129b1177ebf74 (table default.Table_5401cf6f9421453fbe9e59937a098182 [id=93748d777f2243178ee3e2f1532d37b7]) successfully deleted
I20250112 20:51:35.338469 1164 raft_consensus.cc:357] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.338887 1164 raft_consensus.cc:383] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.339036 1164 raft_consensus.cc:738] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.339365 1164 consensus_queue.cc:260] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.340018 1164 ts_tablet_manager.cc:1428] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:35.339913 1821 consensus_queue.cc:1035] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:35.339650 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9c6d5d425e6a493daa46900cbd8db166 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c6d5d425e6a493daa46900cbd8db166 already in progress: creating tablet
I20250112 20:51:35.340698 1164 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.341949 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 420) took 1094 ms. Trace:
W20250112 20:51:35.342931 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e9f669a739d94f99a5b7e9ccccd29545 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9f669a739d94f99a5b7e9ccccd29545 already in progress: creating tablet
I20250112 20:51:35.343461 1185 rpcz_store.cc:276] 0112 20:51:34.247670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.248915 (+ 1245us) service_pool.cc:225] Handling call
0112 20:51:35.341708 (+1092793us) ts_tablet_manager.cc:1145] Deleting tablet 1fd34e829d0645ea80daaff5d27cb266
0112 20:51:35.341716 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.341901 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092569}
I20250112 20:51:35.344022 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 419) took 1096 ms. Trace:
I20250112 20:51:35.344151 1185 rpcz_store.cc:276] 0112 20:51:34.247521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.248560 (+ 1039us) service_pool.cc:225] Handling call
0112 20:51:35.343887 (+1095327us) ts_tablet_manager.cc:1145] Deleting tablet 1fd47d1c1a4b42bdbff97f619dc95134
0112 20:51:35.343895 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.343977 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091879}
I20250112 20:51:35.344696 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 421) took 1033 ms. Trace:
W20250112 20:51:35.344633 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2ab635fec8a64efd8e0c527d913ccd7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ab635fec8a64efd8e0c527d913ccd7d already in progress: creating tablet
I20250112 20:51:35.344969 1185 rpcz_store.cc:276] 0112 20:51:34.311367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.311481 (+ 114us) service_pool.cc:225] Handling call
0112 20:51:35.344583 (+1033102us) ts_tablet_manager.cc:1145] Deleting tablet 8c38c8594a384689ae57c26174765284
0112 20:51:35.344591 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.344650 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026015}
I20250112 20:51:35.345425 1185 tablet_replica.cc:331] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.345933 1185 raft_consensus.cc:2238] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.346340 1185 raft_consensus.cc:2267] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.343389 1164 tablet_bootstrap.cc:654] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:35.346796 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4f34aa32d6474561988b2cf3d0e8fd3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f34aa32d6474561988b2cf3d0e8fd3b already in progress: creating tablet
I20250112 20:51:35.348088 1185 ts_tablet_manager.cc:1905] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:35.348709 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4561956c97824824a55283a8da21929e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4561956c97824824a55283a8da21929e already in progress: creating tablet
W20250112 20:51:35.350419 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2469127891524be5ac3dc13fbcf35c7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2469127891524be5ac3dc13fbcf35c7a already in progress: creating tablet
W20250112 20:51:35.352495 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1fd34e829d0645ea80daaff5d27cb266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd34e829d0645ea80daaff5d27cb266 already in progress: creating tablet
W20250112 20:51:35.354169 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1fd47d1c1a4b42bdbff97f619dc95134 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fd47d1c1a4b42bdbff97f619dc95134 already in progress: creating tablet
I20250112 20:51:35.354833 1185 ts_tablet_manager.cc:1918] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.359090 1185 log.cc:1198] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/201a7e2f17bb46fa918dad8f878a4065
W20250112 20:51:35.358907 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c38c8594a384689ae57c26174765284 because the tablet was not found. No further retry: Not found: Tablet not found: 8c38c8594a384689ae57c26174765284
I20250112 20:51:35.359715 1185 ts_tablet_manager.cc:1939] T 201a7e2f17bb46fa918dad8f878a4065 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.361074 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 423) took 1043 ms. Trace:
I20250112 20:51:35.362097 1185 rpcz_store.cc:276] 0112 20:51:34.317737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.318218 (+ 481us) service_pool.cc:225] Handling call
0112 20:51:35.345348 (+1027130us) ts_tablet_manager.cc:1145] Deleting tablet 201a7e2f17bb46fa918dad8f878a4065
0112 20:51:35.345356 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.351652 (+ 6296us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.354688 (+ 3036us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.361040 (+ 6352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025817}
I20250112 20:51:35.362038 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]) successfully deleted
I20250112 20:51:35.362962 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 425) took 1029 ms. Trace:
I20250112 20:51:35.363395 1185 rpcz_store.cc:276] 0112 20:51:34.333413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.333497 (+ 84us) service_pool.cc:225] Handling call
0112 20:51:35.362815 (+1029318us) ts_tablet_manager.cc:1145] Deleting tablet f76dbd7f681b418ca7bb6c6e8f1f1e58
0112 20:51:35.362823 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.362924 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018898}
I20250112 20:51:35.364187 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 424) took 1032 ms. Trace:
I20250112 20:51:35.364601 1185 rpcz_store.cc:276] 0112 20:51:34.332135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.332183 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:35.364070 (+1031887us) ts_tablet_manager.cc:1145] Deleting tablet 304e1c8abe094bd99a99fd358b697f81
0112 20:51:35.364082 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.364154 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017409}
W20250112 20:51:35.364523 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f3ff8ac6f4d94c10b03a764bbf910c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3ff8ac6f4d94c10b03a764bbf910c44 already in progress: creating tablet
I20250112 20:51:35.365442 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 426) took 1028 ms. Trace:
I20250112 20:51:35.365999 1185 rpcz_store.cc:276] 0112 20:51:34.337012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.337800 (+ 788us) service_pool.cc:225] Handling call
0112 20:51:35.365289 (+1027489us) ts_tablet_manager.cc:1145] Deleting tablet 49b8cc99742f4f9ba68eb1aa17cab248
0112 20:51:35.365298 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.365400 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013094}
I20250112 20:51:35.367594 1185 tablet_replica.cc:331] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.367957 1185 raft_consensus.cc:2238] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.368217 1185 raft_consensus.cc:2267] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.368981 1186 ts_tablet_manager.cc:1918] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.369335 1185 ts_tablet_manager.cc:1905] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:35.369036 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 already in progress: creating tablet
I20250112 20:51:35.370735 1164 tablet_bootstrap.cc:492] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.370921 1164 ts_tablet_manager.cc:1397] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.003s
I20250112 20:51:35.371800 1164 raft_consensus.cc:357] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.372033 1164 raft_consensus.cc:383] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.372118 1164 raft_consensus.cc:738] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.372360 1164 consensus_queue.cc:260] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.372888 1164 ts_tablet_manager.cc:1428] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:35.373231 1164 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:35.373778 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 70653110aec74727acec9d63c2eb0f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70653110aec74727acec9d63c2eb0f05 already in progress: creating tablet
I20250112 20:51:35.375264 1164 tablet_bootstrap.cc:654] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.377004 1186 log.cc:1198] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f249d70b116443b7823e093efeda5cc3
W20250112 20:51:35.376837 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a3ca4d49496e42ddb263264ceef2f27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ca4d49496e42ddb263264ceef2f27a already in progress: creating tablet
I20250112 20:51:35.377486 1185 ts_tablet_manager.cc:1918] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.377672 1185 log.cc:1198] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/70c86ad5f59d431e9b1a1fa890fd35c0
I20250112 20:51:35.378016 1186 ts_tablet_manager.cc:1939] T f249d70b116443b7823e093efeda5cc3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.378170 1185 ts_tablet_manager.cc:1939] T 70c86ad5f59d431e9b1a1fa890fd35c0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.379913 1185 tablet_replica.cc:331] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.380172 1185 raft_consensus.cc:2238] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.380434 1185 raft_consensus.cc:2267] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.380561 1186 tablet_replica.cc:331] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.381059 1185 ts_tablet_manager.cc:1905] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.380650 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e590f9615c4d47abaad1a36ac74b3c80 (table default.Table_4c02bae8e4d64469b992b13dd36c7e81 [id=7dc6e9f691804365a9696db3b26700bd]) successfully deleted
I20250112 20:51:35.382781 1186 raft_consensus.cc:2238] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.382956 1186 raft_consensus.cc:2267] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.383498 1186 ts_tablet_manager.cc:1905] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.382797 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:35.385157 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f249d70b116443b7823e093efeda5cc3 (table default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]) successfully deleted
I20250112 20:51:35.388185 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.387773 1185 ts_tablet_manager.cc:1918] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:35.388540 1185 log.cc:1198] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/decdec0d34344ec4b422643b0d4e7659
I20250112 20:51:35.389061 1185 ts_tablet_manager.cc:1939] T decdec0d34344ec4b422643b0d4e7659 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.389142 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.391455 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.392119 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.392452 1185 tablet_replica.cc:331] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.392690 1185 raft_consensus.cc:2238] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.392974 1185 raft_consensus.cc:2267] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.393452 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.393980 1185 ts_tablet_manager.cc:1905] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.395782 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.396245 738 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.397045 738 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.397121 874 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.397181 884 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.398065 738 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.398414 738 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.398777 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f769533bfde0448792d01b883e41763c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.400008 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 201a7e2f17bb46fa918dad8f878a4065 (table default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]) successfully deleted
W20250112 20:51:35.400511 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 already in progress: creating tablet
W20250112 20:51:35.402050 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 304e1c8abe094bd99a99fd358b697f81 because the tablet was not found. No further retry: Not found: Tablet not found: 304e1c8abe094bd99a99fd358b697f81
W20250112 20:51:35.403003 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 49b8cc99742f4f9ba68eb1aa17cab248 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49b8cc99742f4f9ba68eb1aa17cab248 already in progress: creating tablet
I20250112 20:51:35.403779 1185 ts_tablet_manager.cc:1918] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.403963 1185 log.cc:1198] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:35.404412 1185 ts_tablet_manager.cc:1939] T e590f9615c4d47abaad1a36ac74b3c80 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.405556 1185 tablet_replica.cc:331] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.405815 1185 raft_consensus.cc:2238] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.406097 1185 pending_rounds.cc:62] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:35.406239 1185 pending_rounds.cc:69] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585026601775104 op_type: NO_OP noop_request { }
I20250112 20:51:35.406455 1185 raft_consensus.cc:2883] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.406618 1185 raft_consensus.cc:2267] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.404237 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 669e1cfd407e4d87a63e6832d9c6fa1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 669e1cfd407e4d87a63e6832d9c6fa1f already in progress: creating tablet
I20250112 20:51:35.407570 1185 ts_tablet_manager.cc:1905] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.407560 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (table default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]) successfully deleted
W20250112 20:51:35.412592 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f0daaeb0bf6046da8de969b1f4c71c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0daaeb0bf6046da8de969b1f4c71c84 already in progress: creating tablet
I20250112 20:51:35.414079 1186 ts_tablet_manager.cc:1918] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:35.414016 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f3ff8ac6f4d94c10b03a764bbf910c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3ff8ac6f4d94c10b03a764bbf910c44 already in progress: creating tablet
I20250112 20:51:35.414778 1186 log.cc:1198] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/23d5269a5c8c4c049265e18bff4e2c45
I20250112 20:51:35.414914 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_05c8c0dc76884c47b0e87e2801dffe72 [id=2f68c758f6554102961912265cf48921]) successfully deleted
I20250112 20:51:35.415164 1185 ts_tablet_manager.cc:1918] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.415300 1185 log.cc:1198] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/af1aac31fd32400b9094f8e2de4fe42c
I20250112 20:51:35.415577 1745 raft_consensus.cc:491] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.415679 1186 ts_tablet_manager.cc:1939] T 23d5269a5c8c4c049265e18bff4e2c45 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.415752 1745 raft_consensus.cc:513] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.415941 1185 ts_tablet_manager.cc:1939] T af1aac31fd32400b9094f8e2de4fe42c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:35.416178 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 already in progress: creating tablet
I20250112 20:51:35.416499 1745 leader_election.cc:290] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.417284 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:35.417399 1185 tablet_replica.cc:331] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:35.417171 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 70653110aec74727acec9d63c2eb0f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70653110aec74727acec9d63c2eb0f05 already in progress: creating tablet
I20250112 20:51:35.417582 1185 raft_consensus.cc:2238] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.417858 1185 pending_rounds.cc:62] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:35.417976 1185 pending_rounds.cc:69] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585027104366592 op_type: NO_OP noop_request { }
I20250112 20:51:35.418160 1185 raft_consensus.cc:2883] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.418321 1185 raft_consensus.cc:2267] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.418215 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a3ca4d49496e42ddb263264ceef2f27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ca4d49496e42ddb263264ceef2f27a already in progress: creating tablet
I20250112 20:51:35.419304 1185 ts_tablet_manager.cc:1905] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.419211 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e590f9615c4d47abaad1a36ac74b3c80 (table default.Table_4c02bae8e4d64469b992b13dd36c7e81 [id=7dc6e9f691804365a9696db3b26700bd]) successfully deleted
I20250112 20:51:35.419807 1186 tablet_replica.cc:331] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.420042 2028 tablet.cc:1721] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.420043 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet af1aac31fd32400b9094f8e2de4fe42c (table default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]) successfully deleted
I20250112 20:51:35.420722 1186 raft_consensus.cc:2238] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:35.421140 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ac4e304a1212412f8e99bfd222a110b5 because the tablet was not found. No further retry: Not found: Tablet not found: ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:35.421752 1186 raft_consensus.cc:2267] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.421988 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 23d5269a5c8c4c049265e18bff4e2c45 (table default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]) successfully deleted
I20250112 20:51:35.422948 1186 ts_tablet_manager.cc:1905] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:35.422860 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet bc5af0bc83bc415b972cbbe0fb39f75f because the tablet was not found. No further retry: Not found: Tablet not found: bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:35.432411 1185 ts_tablet_manager.cc:1918] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.433025 1185 log.cc:1198] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/23d5269a5c8c4c049265e18bff4e2c45
I20250112 20:51:35.433609 1185 ts_tablet_manager.cc:1939] T 23d5269a5c8c4c049265e18bff4e2c45 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.434805 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 23d5269a5c8c4c049265e18bff4e2c45 (table default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]) successfully deleted
I20250112 20:51:35.440312 1185 tablet_replica.cc:331] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:35.440765 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 00c0712ab2de43eea8c77ab39a971a2c because the tablet was not found. No further retry: Not found: Tablet not found: 00c0712ab2de43eea8c77ab39a971a2c
W20250112 20:51:35.441035 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1d97ac93467a4a7a8ae44d52719f191c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d97ac93467a4a7a8ae44d52719f191c already in progress: creating tablet
W20250112 20:51:35.441233 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1d60604713524dba8f3a42c4f3bc7300 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d60604713524dba8f3a42c4f3bc7300 already in progress: creating tablet
W20250112 20:51:35.441409 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9570269368e14b1abeaffdcc741c6f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9570269368e14b1abeaffdcc741c6f3a already in progress: creating tablet
W20250112 20:51:35.441633 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d17e6d263014448b96c5d1f03013e6e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17e6d263014448b96c5d1f03013e6e7 already in progress: creating tablet
W20250112 20:51:35.441859 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d7117edf078848c0a1c83ab8e804dfed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7117edf078848c0a1c83ab8e804dfed already in progress: creating tablet
I20250112 20:51:35.443487 1185 raft_consensus.cc:2238] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.443796 1185 pending_rounds.cc:62] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:35.443907 1185 pending_rounds.cc:69] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585030143606784 op_type: NO_OP noop_request { }
I20250112 20:51:35.444082 1185 raft_consensus.cc:2883] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.444242 1185 raft_consensus.cc:2267] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.444820 2029 tablet.cc:1721] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.445986 1185 ts_tablet_manager.cc:1905] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.452817 1186 ts_tablet_manager.cc:1918] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.452948 1186 log.cc:1198] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b84624dc685b401192ca753ff59da016
I20250112 20:51:35.453318 1186 ts_tablet_manager.cc:1939] T b84624dc685b401192ca753ff59da016 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.454174 1186 tablet_replica.cc:331] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.454236 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b84624dc685b401192ca753ff59da016 (table default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]) successfully deleted
I20250112 20:51:35.456472 1164 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.456743 1164 ts_tablet_manager.cc:1397] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.009s
I20250112 20:51:35.458957 1164 raft_consensus.cc:357] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.459404 1164 raft_consensus.cc:383] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.459658 1164 raft_consensus.cc:738] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.460022 1164 consensus_queue.cc:260] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.463804 1186 raft_consensus.cc:2238] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:35.464284 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dc920e92de545fc961f82e7d0de28cd consensus state has no leader
I20250112 20:51:35.464560 1186 pending_rounds.cc:62] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.464718 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e0fd0e3995274cd9b07af514ac3ef3cd" } new_table_name: "default.Table_0787fb4223774e1889d1d09fb80f87f5" modify_external_catalogs: true
I20250112 20:51:35.464988 1186 pending_rounds.cc:69] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585026601775104 op_type: NO_OP noop_request { }
I20250112 20:51:35.465435 1186 raft_consensus.cc:2883] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.465839 1186 raft_consensus.cc:2267] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.466558 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.467249 1186 ts_tablet_manager.cc:1905] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:35.467892 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3facda2ee004b2e9487194ba040c9cf consensus state has no leader
I20250112 20:51:35.468163 449 master_service.cc:946] ReplaceTablet: received request to replace tablet a48df2ec556d43a3b22129b1177ebf74 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.468741 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_743de41614e248bd9cffca489bc49570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:35.467378 2029 tablet.cc:1716] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Skipping requested alter to schema version 2, tablet already version 2
W20250112 20:51:35.469321 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_743de41614e248bd9cffca489bc49570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.461295 2028 tablet.cc:1716] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Skipping requested alter to schema version 2, tablet already version 2
I20250112 20:51:35.465193 1185 ts_tablet_manager.cc:1918] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.470714 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_341aa7fe62034251879c0e39f76dd6e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:35.470981 1185 log.cc:1198] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d8330fa1e4654a1a9f167a1120a103fc
W20250112 20:51:35.471215 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_341aa7fe62034251879c0e39f76dd6e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.471505 1185 ts_tablet_manager.cc:1939] T d8330fa1e4654a1a9f167a1120a103fc P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:35.473495 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 008d69d199d64bdf85da17d8d3bf6264 has no consensus state
I20250112 20:51:35.473693 1164 ts_tablet_manager.cc:1428] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250112 20:51:35.473804 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d8330fa1e4654a1a9f167a1120a103fc (table default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]) successfully deleted
I20250112 20:51:35.473891 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0b49521c49204a03b257a7b38a4132c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:35.474089 1164 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:35.474103 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e72fda1743fc4f96b29add6e36e884de because the tablet was not found. No further retry: Not found: Tablet not found: e72fda1743fc4f96b29add6e36e884de
W20250112 20:51:35.474308 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f769533bfde0448792d01b883e41763c because the tablet was not found. No further retry: Not found: Tablet not found: f769533bfde0448792d01b883e41763c
W20250112 20:51:35.474359 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b49521c49204a03b257a7b38a4132c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.476256 1164 tablet_bootstrap.cc:654] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.478823 452 catalog_manager.cc:5526] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:35.479277 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:35.479606 452 catalog_manager.cc:5526] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:35.480188 452 catalog_manager.cc:5526] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:35.482458 1178 tablet_bootstrap.cc:492] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.482643 1178 ts_tablet_manager.cc:1397] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.261s user 0.014s sys 0.041s
I20250112 20:51:35.483449 1178 raft_consensus.cc:357] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.483661 1178 raft_consensus.cc:383] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.483728 1178 raft_consensus.cc:738] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.483919 1178 consensus_queue.cc:260] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.484647 1178 ts_tablet_manager.cc:1428] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:35.484937 1178 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.489226 1178 tablet_bootstrap.cc:654] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.490796 1186 ts_tablet_manager.cc:1918] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.490911 1186 log.cc:1198] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/af1aac31fd32400b9094f8e2de4fe42c
I20250112 20:51:35.491281 1186 ts_tablet_manager.cc:1939] T af1aac31fd32400b9094f8e2de4fe42c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.492146 1186 tablet_replica.cc:331] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.492293 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet af1aac31fd32400b9094f8e2de4fe42c (table default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]) successfully deleted
I20250112 20:51:35.492350 1186 raft_consensus.cc:2238] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.492548 1186 pending_rounds.cc:62] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.492640 1186 pending_rounds.cc:69] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585028687822848 op_type: NO_OP noop_request { }
W20250112 20:51:35.495468 847 consensus_peers.cc:487] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:35.492739 1186 raft_consensus.cc:2883] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.496536 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9989f3dc61df4f75b9bdb85ba3ca3f37
I20250112 20:51:35.496536 1186 raft_consensus.cc:2267] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.497186 1184 ts_tablet_manager.cc:1918] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.497398 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.497712 1186 ts_tablet_manager.cc:1905] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.497980 453 catalog_manager.cc:6596] ReplaceTablet: tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 deleted and replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6
I20250112 20:51:35.497749 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:35.498852 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 already in progress: creating tablet
I20250112 20:51:35.499207 1184 log.cc:1198] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/00c0712ab2de43eea8c77ab39a971a2c
I20250112 20:51:35.499706 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.500002 1184 ts_tablet_manager.cc:1939] T 00c0712ab2de43eea8c77ab39a971a2c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.501065 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64cd83b199bb4da28d39da449945c389" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.503554 710 leader_election.cc:304] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.504668 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.504865 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.505164 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.505475 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.512660 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_06d8dc6fe3af4be68fee565a90654657"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.513286 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06d8dc6fe3af4be68fee565a90654657 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.514750 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e5db37687d384be894f7f6239198cd82" } new_table_name: "default.Table_0a377dabb6b14129ba1e8519d731bafb" modify_external_catalogs: true
I20250112 20:51:35.516629 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.517675 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 420) took 1174 ms. Trace:
I20250112 20:51:35.517796 1184 rpcz_store.cc:276] 0112 20:51:34.326645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.326882 (+ 237us) service_pool.cc:225] Handling call
0112 20:51:35.242429 (+915547us) ts_tablet_manager.cc:1145] Deleting tablet 00c0712ab2de43eea8c77ab39a971a2c
0112 20:51:35.242667 (+ 238us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.358082 (+115415us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.485735 (+127653us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.500915 (+ 15180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":897690}
I20250112 20:51:35.518131 1184 tablet_replica.cc:331] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.518301 1184 raft_consensus.cc:2238] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.518651 1184 raft_consensus.cc:2267] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.519222 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 00c0712ab2de43eea8c77ab39a971a2c (table default.Table_4ffedc1a9fee40b4bf3a93f178099ceb [id=9b34a053321c427cb2ffccb8a81b57a6]) successfully deleted
I20250112 20:51:35.519773 1184 ts_tablet_manager.cc:1905] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.522825 1931 raft_consensus.cc:2743] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.527081 1184 ts_tablet_manager.cc:1918] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.527264 1184 log.cc:1198] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3a2ee62fff7d4cf7b4aaeacf63ed8e02
I20250112 20:51:35.527736 1184 ts_tablet_manager.cc:1939] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.529430 1186 ts_tablet_manager.cc:1918] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.529430 1932 consensus_queue.cc:1035] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.530314 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 421) took 1202 ms. Trace:
I20250112 20:51:35.531016 1184 rpcz_store.cc:276] 0112 20:51:34.327482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.329900 (+ 2418us) service_pool.cc:225] Handling call
0112 20:51:35.518047 (+1188147us) ts_tablet_manager.cc:1145] Deleting tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02
0112 20:51:35.518057 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.523564 (+ 5507us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.526964 (+ 3400us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.530265 (+ 3301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172063}
I20250112 20:51:35.532456 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 (table default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]) successfully deleted
I20250112 20:51:35.532858 1184 tablet_replica.cc:331] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.533238 1186 log.cc:1198] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/83c13126e53b46c0a1e7beafe1b0d295
I20250112 20:51:35.534195 1186 ts_tablet_manager.cc:1939] T 83c13126e53b46c0a1e7beafe1b0d295 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.534585 1184 raft_consensus.cc:2238] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.535724 1184 raft_consensus.cc:2267] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.537369 1186 tablet_replica.cc:331] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.537561 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 83c13126e53b46c0a1e7beafe1b0d295 (table default.Table_c62b92e5784e44af9e27e902aa48a6ca [id=212d41ddb2d248a0bcf3aa02f7b10bcd]) successfully deleted
I20250112 20:51:35.539172 1184 ts_tablet_manager.cc:1905] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.539767 1942 raft_consensus.cc:491] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.545315 1942 raft_consensus.cc:513] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.546046 1942 leader_election.cc:290] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.546447 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f636df32bd34438bb54989736eaa431" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.546512 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f636df32bd34438bb54989736eaa431" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.546716 770 raft_consensus.cc:2463] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:35.546792 1163 tablet_bootstrap.cc:492] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.546931 1163 ts_tablet_manager.cc:1397] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.257s user 0.038s sys 0.020s
I20250112 20:51:35.548012 1163 raft_consensus.cc:357] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.548128 1745 consensus_queue.cc:1035] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.548274 1163 raft_consensus.cc:383] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.548358 1163 raft_consensus.cc:738] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.548569 1163 consensus_queue.cc:260] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.549175 1163 ts_tablet_manager.cc:1428] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:35.549347 1186 raft_consensus.cc:2238] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.549697 1163 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.551367 1163 tablet_bootstrap.cc:654] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:35.549376 847 leader_election.cc:343] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.551623 847 leader_election.cc:304] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:35.552143 1942 raft_consensus.cc:2798] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:35.552234 1942 raft_consensus.cc:491] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:35.552335 1942 raft_consensus.cc:3054] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:35.554139 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4ef01b41b6140d6bdad07bc79dd7dc5 consensus state has no leader
I20250112 20:51:35.554545 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0bb556ec81044d1f8296628212fc4875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.555058 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb556ec81044d1f8296628212fc4875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.555047 1942 raft_consensus.cc:513] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.557073 1942 leader_election.cc:290] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.557137 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f636df32bd34438bb54989736eaa431" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:35.557348 770 raft_consensus.cc:3054] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.557521 1186 raft_consensus.cc:2267] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.556720 1824 raft_consensus.cc:491] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.557976 1824 raft_consensus.cc:513] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.558842 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f636df32bd34438bb54989736eaa431" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:35.559141 1824 leader_election.cc:290] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:35.559767 847 leader_election.cc:343] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.559808 1164 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.559959 1164 ts_tablet_manager.cc:1397] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.005s
W20250112 20:51:35.560115 983 leader_election.cc:343] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b84624dc685b401192ca753ff59da016
I20250112 20:51:35.560196 770 raft_consensus.cc:2463] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:35.560638 847 leader_election.cc:304] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:35.561091 1942 raft_consensus.cc:2798] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:35.561050 1164 raft_consensus.cc:357] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.561237 1942 raft_consensus.cc:695] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:35.561259 1164 raft_consensus.cc:383] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.561336 1164 raft_consensus.cc:738] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.561475 1942 consensus_queue.cc:237] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.561573 1164 consensus_queue.cc:260] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.562132 1164 ts_tablet_manager.cc:1428] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:35.564178 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 9880e7d303b54d959bca4e8464a6b30c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.565008 1186 ts_tablet_manager.cc:1905] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.569618 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]). Expected version 1 got 0
I20250112 20:51:35.570710 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8aaa5246382745ecbea4b9bd4825e8f1
I20250112 20:51:35.572762 1164 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.573676 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a7e2f17bb46fa918dad8f878a4065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.574019 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.574568 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.574627 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3f359655c94f1d905943069548845b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.574867 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.575680 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.575773 874 tablet_service.cc:1514] Processing DeleteTablet for tablet d770e0399e80491f8de9b08d1eb2f139 with delete_type TABLET_DATA_DELETED (replaced by tablet 3343e1d7f1d04a498bd0ee00311b6a61) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.575942 1185 tablet_replica.cc:331] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.575946 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.575994 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e590f9615c4d47abaad1a36ac74b3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.576156 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c38c8594a384689ae57c26174765284 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:26 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.576162 1185 raft_consensus.cc:2238] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.576160 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d738720dd7414764b4d1191cdc37895d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.576305 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.576364 1185 raft_consensus.cc:2267] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.576431 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.576520 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e72fda1743fc4f96b29add6e36e884de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.577659 1185 ts_tablet_manager.cc:1905] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.579319 1163 tablet_bootstrap.cc:492] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.579466 1163 ts_tablet_manager.cc:1397] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20250112 20:51:35.580231 1163 raft_consensus.cc:357] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.580468 1163 raft_consensus.cc:383] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.580554 1163 raft_consensus.cc:738] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.580765 1163 consensus_queue.cc:260] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.583173 1164 tablet_bootstrap.cc:654] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.585482 1186 ts_tablet_manager.cc:1918] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.584398 1185 ts_tablet_manager.cc:1918] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:35.585652 1185 log.cc:1198] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8aaa5246382745ecbea4b9bd4825e8f1
I20250112 20:51:35.586068 1185 ts_tablet_manager.cc:1939] T 8aaa5246382745ecbea4b9bd4825e8f1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.587554 1184 ts_tablet_manager.cc:1918] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.589184 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_15e3fd8e71984a81acb2dabf8f3563f7 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]) successfully deleted
I20250112 20:51:35.591463 1184 log.cc:1198] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:35.591979 1184 ts_tablet_manager.cc:1939] T d770e0399e80491f8de9b08d1eb2f139 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.590436 1163 ts_tablet_manager.cc:1428] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250112 20:51:35.592993 1163 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.594660 1163 tablet_bootstrap.cc:654] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.594830 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84624dc685b401192ca753ff59da016" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.595108 770 raft_consensus.cc:2463] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:35.591325 1178 tablet_bootstrap.cc:492] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.595687 1178 ts_tablet_manager.cc:1397] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.005s
I20250112 20:51:35.596453 1178 raft_consensus.cc:357] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.596645 1178 raft_consensus.cc:383] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.596700 1178 raft_consensus.cc:738] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.596990 1178 consensus_queue.cc:260] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.599225 1178 ts_tablet_manager.cc:1428] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:35.600561 1178 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.600724 1186 log.cc:1198] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/64cd83b199bb4da28d39da449945c389
I20250112 20:51:35.601099 1186 ts_tablet_manager.cc:1939] T 64cd83b199bb4da28d39da449945c389 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.603945 1178 tablet_bootstrap.cc:654] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.604205 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.595490 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 422) took 1205 ms. Trace:
I20250112 20:51:35.596549 983 leader_election.cc:304] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.604868 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605028 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaa5246382745ecbea4b9bd4825e8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605115 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605173 1824 raft_consensus.cc:2798] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:35.605160 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3407e578085546e59a593ea9e2016ebf" } modify_external_catalogs: true
I20250112 20:51:35.605249 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605276 1824 raft_consensus.cc:491] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:35.605329 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c13126e53b46c0a1e7beafe1b0d295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605366 1824 raft_consensus.cc:3054] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.604215 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.605360 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3407e578085546e59a593ea9e2016ebf" } modify_external_catalogs: true
I20250112 20:51:35.606238 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6e73aeaec7af4cd0952e8f427103f61f" } modify_external_catalogs: true
I20250112 20:51:35.606377 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6e73aeaec7af4cd0952e8f427103f61f" } modify_external_catalogs: true
I20250112 20:51:35.606956 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a18054e19f524ebd88965aa3727a35f8" } modify_external_catalogs: true
I20250112 20:51:35.607095 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a18054e19f524ebd88965aa3727a35f8" } modify_external_catalogs: true
I20250112 20:51:35.607894 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:35.608341 449 catalog_manager.cc:6596] ReplaceTablet: tablet 9880e7d303b54d959bca4e8464a6b30c deleted and replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0
I20250112 20:51:35.608291 1824 raft_consensus.cc:513] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.608704 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.609827 1824 leader_election.cc:290] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:35.610544 983 leader_election.cc:343] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b84624dc685b401192ca753ff59da016
I20250112 20:51:35.610783 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.610770 1184 rpcz_store.cc:276] 0112 20:51:34.389528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.389585 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:35.532739 (+1143154us) ts_tablet_manager.cc:1145] Deleting tablet d770e0399e80491f8de9b08d1eb2f139
0112 20:51:35.532763 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.543637 (+ 10874us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.587451 (+ 43814us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.595430 (+ 7979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138956}
I20250112 20:51:35.611346 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.611655 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d770e0399e80491f8de9b08d1eb2f139 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
I20250112 20:51:35.612635 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0ede004124b94194a25701ee39ebfc60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.613193 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ede004124b94194a25701ee39ebfc60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.613020 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 426) took 1221 ms. Trace:
I20250112 20:51:35.613433 1184 rpcz_store.cc:276] 0112 20:51:34.391719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.394725 (+ 3006us) service_pool.cc:225] Handling call
0112 20:51:35.611022 (+1216297us) ts_tablet_manager.cc:1145] Deleting tablet 669e1cfd407e4d87a63e6832d9c6fa1f
0112 20:51:35.611033 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.612970 (+ 1937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196536}
I20250112 20:51:35.613797 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 423) took 1222 ms. Trace:
I20250112 20:51:35.613873 1184 rpcz_store.cc:276] 0112 20:51:34.391392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.394762 (+ 3370us) service_pool.cc:225] Handling call
0112 20:51:35.613655 (+1218893us) ts_tablet_manager.cc:1145] Deleting tablet bc5af0bc83bc415b972cbbe0fb39f75f
0112 20:51:35.613663 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.613728 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196415}
I20250112 20:51:35.614099 1184 tablet_replica.cc:331] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.614262 1184 raft_consensus.cc:2238] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.614986 1185 tablet_replica.cc:331] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:35.615056 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84624dc685b401192ca753ff59da016" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:35.615249 770 raft_consensus.cc:3054] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.615518 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 64cd83b199bb4da28d39da449945c389 (table default.Table_0afc27e58eec42daadf528a66340c624 [id=370444777ab4411f837c19bc3824376a]) successfully deleted
W20250112 20:51:35.617911 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 669e1cfd407e4d87a63e6832d9c6fa1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 669e1cfd407e4d87a63e6832d9c6fa1f already in progress: creating tablet
W20250112 20:51:35.618207 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet bc5af0bc83bc415b972cbbe0fb39f75f because the tablet was not found. No further retry: Not found: Tablet not found: bc5af0bc83bc415b972cbbe0fb39f75f
I20250112 20:51:35.618350 770 raft_consensus.cc:2463] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:35.619021 1185 raft_consensus.cc:2238] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.619345 1185 raft_consensus.cc:2267] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.619309 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0787fb4223774e1889d1d09fb80f87f5" } modify_external_catalogs: true
I20250112 20:51:35.619858 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0787fb4223774e1889d1d09fb80f87f5" } modify_external_catalogs: true
I20250112 20:51:35.623878 1982 consensus_queue.cc:1035] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.624715 1186 tablet_replica.cc:331] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.625111 1186 raft_consensus.cc:2238] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.625557 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet efa7f6a1e9804e6ab822f9d4c004b1b3
W20250112 20:51:35.625406 847 consensus_peers.cc:487] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:35.626828 847 consensus_peers.cc:487] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aaa5246382745ecbea4b9bd4825e8f1. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:35.627111 1186 raft_consensus.cc:2267] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.628248 1186 ts_tablet_manager.cc:1905] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.620949 983 leader_election.cc:304] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.630280 1824 raft_consensus.cc:2798] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:35.630358 1185 ts_tablet_manager.cc:1905] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.630463 1824 raft_consensus.cc:695] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:35.630962 1824 consensus_queue.cc:237] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:35.634341 847 consensus_peers.cc:487] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c14102aecb743e2a82660039bd0f038. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:35.634780 886 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.635507 1184 raft_consensus.cc:2267] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.635511 738 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.636093 1184 ts_tablet_manager.cc:1905] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.638149 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.641341 738 tablet_service.cc:1467] Processing CreateTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 (DEFAULT_TABLE table=default.Table_9ced728bcaeb4a668b3c1cc24916d012 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.641759 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30a34c1abcb4f72a7f4f8a8f5be3720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.642016 886 tablet_service.cc:1467] Processing CreateTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 (DEFAULT_TABLE table=default.Table_9ced728bcaeb4a668b3c1cc24916d012 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.642244 1017 tablet_service.cc:1467] Processing CreateTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 (DEFAULT_TABLE table=default.Table_9ced728bcaeb4a668b3c1cc24916d012 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.642378 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30a34c1abcb4f72a7f4f8a8f5be3720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.642590 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30a34c1abcb4f72a7f4f8a8f5be3720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.646972 749 tablet_service.cc:1467] Processing CreateTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 (DEFAULT_TABLE table=default.Table_8c9a7a06fb384d4aa61879e57247d36e [id=ce81dff010874269954c90b958f031ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.647393 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.648577 886 tablet_service.cc:1467] Processing CreateTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 (DEFAULT_TABLE table=default.Table_8c9a7a06fb384d4aa61879e57247d36e [id=ce81dff010874269954c90b958f031ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.649056 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.649026 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 (DEFAULT_TABLE table=default.Table_8c9a7a06fb384d4aa61879e57247d36e [id=ce81dff010874269954c90b958f031ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.649416 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.649641 752 tablet_service.cc:1467] Processing CreateTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.649972 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d341f6ac6848d9a8e60f689dd24a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.655483 1745 raft_consensus.cc:491] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.655647 1745 raft_consensus.cc:513] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.656003 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 267ebb3c30194d9fa6a092d9bec4eadf from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.656236 1745 leader_election.cc:290] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.656641 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:35.656842 912 raft_consensus.cc:2463] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:35.657286 710 leader_election.cc:304] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:35.657615 1745 raft_consensus.cc:2798] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:35.657711 1745 raft_consensus.cc:491] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:35.657832 1745 raft_consensus.cc:3054] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.656746 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5d167065029a46b791d413db8a2c4bcc
I20250112 20:51:35.662070 1745 raft_consensus.cc:513] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.662307 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 921ae136d5e2438e950b96f156f23da3
I20250112 20:51:35.662328 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.662582 1048 raft_consensus.cc:2463] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:35.663388 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.663607 1824 raft_consensus.cc:491] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.663692 1745 leader_election.cc:290] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.663803 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d341f6ac6848d9a8e60f689dd24a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.663779 1824 raft_consensus.cc:513] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.664999 1824 leader_election.cc:290] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.665066 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deb6e37f75e4aebacb04c4553049940" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:35.665263 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:35.665398 912 raft_consensus.cc:3054] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.665669 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deb6e37f75e4aebacb04c4553049940" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.665884 770 raft_consensus.cc:2463] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:35.667812 983 leader_election.cc:304] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:35.668373 983 leader_election.cc:343] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.668941 912 raft_consensus.cc:2463] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:35.669379 885 tablet_service.cc:1467] Processing CreateTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 (DEFAULT_TABLE table=default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.669407 710 leader_election.cc:304] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:35.669731 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 008d69d199d64bdf85da17d8d3bf6264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.669915 1824 raft_consensus.cc:2798] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:35.669979 749 tablet_service.cc:1467] Processing CreateTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 (DEFAULT_TABLE table=default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.670058 1824 raft_consensus.cc:491] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:35.670006 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ba8529a287434b9ba78e0b6fa7817e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:35.670188 1824 raft_consensus.cc:3054] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.670208 1048 raft_consensus.cc:3054] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.670476 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 008d69d199d64bdf85da17d8d3bf6264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.672441 1745 raft_consensus.cc:2798] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:35.672705 1745 raft_consensus.cc:695] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:35.673051 1745 consensus_queue.cc:237] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.674294 884 tablet_service.cc:1467] Processing CreateTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 (DEFAULT_TABLE table=default.Table_455466a0838742f3a2cc35cdab729433 [id=fada45a1d9a64ff4871c139b8767f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.674719 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868ce9397b884b0095dae39ea57f5fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.674933 738 tablet_service.cc:1467] Processing CreateTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 (DEFAULT_TABLE table=default.Table_455466a0838742f3a2cc35cdab729433 [id=fada45a1d9a64ff4871c139b8767f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.675238 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868ce9397b884b0095dae39ea57f5fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.675429 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 (DEFAULT_TABLE table=default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.675777 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 008d69d199d64bdf85da17d8d3bf6264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.679221 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0] alter complete (version 2)
I20250112 20:51:35.682348 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 (DEFAULT_TABLE table=default.Table_7038f1cfb0f944378fdf6ab8daa15d60 [id=e2d5f4685f49452eb6574236b4fb0c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.682915 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3994d41a7ecd418a98a3c203add0d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.684051 886 tablet_service.cc:1467] Processing CreateTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.685788 1048 raft_consensus.cc:2463] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:35.686251 1942 raft_consensus.cc:491] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.688177 1942 raft_consensus.cc:513] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.688040 1824 raft_consensus.cc:513] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.688807 1824 leader_election.cc:290] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.689925 1942 leader_election.cc:290] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.689889 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe28f23448c47d3979f1982f939e39c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.690027 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe28f23448c47d3979f1982f939e39c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:35.690239 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deb6e37f75e4aebacb04c4553049940" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:35.690663 770 raft_consensus.cc:3054] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:35.690852 847 leader_election.cc:343] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:35.691038 847 leader_election.cc:343] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.691136 847 leader_election.cc:304] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:35.691965 882 tablet_service.cc:1467] Processing CreateTablet for tablet f425779195594599a3ab8c4dc5bee7f9 (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.692318 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f425779195594599a3ab8c4dc5bee7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.692930 753 tablet_service.cc:1467] Processing CreateTablet for tablet f425779195594599a3ab8c4dc5bee7f9 (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.693027 756 tablet_service.cc:1467] Processing CreateTablet for tablet 68b4b312c558437892388344767f0ed4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.693307 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f425779195594599a3ab8c4dc5bee7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.693508 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b4b312c558437892388344767f0ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.694216 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.691910 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7deb6e37f75e4aebacb04c4553049940" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:35.694908 983 leader_election.cc:343] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:35.695304 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 (DEFAULT_TABLE table=default.Table_7038f1cfb0f944378fdf6ab8daa15d60 [id=e2d5f4685f49452eb6574236b4fb0c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.695683 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3994d41a7ecd418a98a3c203add0d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.697047 874 tablet_service.cc:1467] Processing CreateTablet for tablet 68b4b312c558437892388344767f0ed4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.697108 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.697142 739 tablet_service.cc:1467] Processing CreateTablet for tablet 05550152671e4e28829b5b102b59f715 (DEFAULT_TABLE table=default.Table_c06477d484e0483589b0530cf5787b9c [id=4b364d74bdc149a09a4765f2c125d35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.697384 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b4b312c558437892388344767f0ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.697649 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05550152671e4e28829b5b102b59f715. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.699707 1942 raft_consensus.cc:2743] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.700165 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d341f6ac6848d9a8e60f689dd24a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.700606 1821 raft_consensus.cc:491] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.700788 1821 raft_consensus.cc:513] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.701432 1821 leader_election.cc:290] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:35.702315 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 (DEFAULT_TABLE table=default.Table_7038f1cfb0f944378fdf6ab8daa15d60 [id=e2d5f4685f49452eb6574236b4fb0c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.706128 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 720aa779aada46d5a888d5011804a6a8 (DEFAULT_TABLE table=default.Table_eb174dc18b7d43d69147c2d69b5f458d [id=ff62e1f5b1aa45408d49878b0c050510]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:35.706413 983 leader_election.cc:343] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:35.706900 885 tablet_service.cc:1467] Processing CreateTablet for tablet 720aa779aada46d5a888d5011804a6a8 (DEFAULT_TABLE table=default.Table_eb174dc18b7d43d69147c2d69b5f458d [id=ff62e1f5b1aa45408d49878b0c050510]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:35.706962 983 leader_election.cc:343] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:35.707069 983 leader_election.cc:304] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:35.707221 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720aa779aada46d5a888d5011804a6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.708245 1011 tablet_service.cc:1467] Processing CreateTablet for tablet f425779195594599a3ab8c4dc5bee7f9 (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.708657 770 raft_consensus.cc:2463] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:35.709173 983 leader_election.cc:304] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.709298 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 68b4b312c558437892388344767f0ed4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.709580 1821 raft_consensus.cc:2743] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.709775 1821 raft_consensus.cc:2798] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:35.709873 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.709923 1821 raft_consensus.cc:695] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:35.710158 1821 consensus_queue.cc:237] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.709772 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet b84624dc685b401192ca753ff59da016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.712791 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4e304a1212412f8e99bfd222a110b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c5b10576bd34af5b02f8d8dcc3759ab) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.714301 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b4b312c558437892388344767f0ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.716207 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b4ef01b41b6140d6bdad07bc79dd7dc5
I20250112 20:51:35.716552 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.717087 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f425779195594599a3ab8c4dc5bee7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.715657 2040 tablet.cc:1721] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.721278 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720aa779aada46d5a888d5011804a6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.722481 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:35.722959 452 catalog_manager.cc:6596] ReplaceTablet: tablet 267ebb3c30194d9fa6a092d9bec4eadf deleted and replaced by tablet f7b722b0eb5944a4b344a3833905a2b0
I20250112 20:51:35.724496 1164 tablet_bootstrap.cc:492] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:35.724638 1164 ts_tablet_manager.cc:1397] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.161s user 0.002s sys 0.007s
I20250112 20:51:35.725648 1164 raft_consensus.cc:357] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.725895 1164 raft_consensus.cc:383] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.726004 1164 raft_consensus.cc:738] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.726285 1164 consensus_queue.cc:260] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.726620 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.726917 1164 ts_tablet_manager.cc:1428] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:35.726964 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.727203 1164 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:35.728193 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.730253 1185 ts_tablet_manager.cc:1918] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.730387 1185 log.cc:1198] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b84624dc685b401192ca753ff59da016
I20250112 20:51:35.730773 1185 ts_tablet_manager.cc:1939] T b84624dc685b401192ca753ff59da016 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:35.732674 1816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:35.735580 1816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:35.738987 1816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:35.740006 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f9f2cdcb247848539893c00bfbef8b18" } new_table_name: "default.Table_a1454445414a4f2882deedb81bad67bf" modify_external_catalogs: true
I20250112 20:51:35.741793 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ca3bce77c81b44b98c37ce00f1536286"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.742463 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca3bce77c81b44b98c37ce00f1536286 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.742861 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_557ef80f7a9c4980aa3aa81e9000ddf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.744376 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_557ef80f7a9c4980aa3aa81e9000ddf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.745043 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e754c35aa13145f18490d18de677c33d" } new_table_name: "default.Table_90f38fee7fd8445a98f6af6d299318bc" modify_external_catalogs: true
I20250112 20:51:35.747017 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_830c03785e1a464fb83fcb19bcd333e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.747730 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_830c03785e1a464fb83fcb19bcd333e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.750411 1178 tablet_bootstrap.cc:492] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.750622 1178 ts_tablet_manager.cc:1397] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.150s user 0.005s sys 0.004s
W20250112 20:51:35.752008 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ca8fc4a45fe448b919ac001a7980deb consensus state has no leader
I20250112 20:51:35.751935 1178 raft_consensus.cc:357] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.752277 1178 raft_consensus.cc:383] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.752460 1178 raft_consensus.cc:738] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.752815 1178 consensus_queue.cc:260] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.687108 1186 ts_tablet_manager.cc:1918] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.754568 1186 log.cc:1198] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3a2ee62fff7d4cf7b4aaeacf63ed8e02
I20250112 20:51:35.755486 1186 ts_tablet_manager.cc:1939] T 3a2ee62fff7d4cf7b4aaeacf63ed8e02 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.757552 1186 tablet_replica.cc:331] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.758363 1186 raft_consensus.cc:2238] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.758817 1186 pending_rounds.cc:62] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.758927 1186 pending_rounds.cc:69] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585027081236480 op_type: NO_OP noop_request { }
I20250112 20:51:35.759395 1186 raft_consensus.cc:2883] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.759611 1186 raft_consensus.cc:2267] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.759841 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ced728bcaeb4a668b3c1cc24916d012" } new_table_name: "default.Table_11a4b78b73c54798b358bc7647928f4e" modify_external_catalogs: true
I20250112 20:51:35.760810 1186 ts_tablet_manager.cc:1905] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:35.770259 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4303e4aec9694c49aca7c1532eaab3ed consensus state has no leader
I20250112 20:51:35.767985 1178 ts_tablet_manager.cc:1428] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20250112 20:51:35.772447 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c55b922c39545ac9c89115cd587ea8c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.773147 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_037b9792e05d4bd08c1ae042e95cba5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.773680 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_037b9792e05d4bd08c1ae042e95cba5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:35.778286 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b30a34c1abcb4f72a7f4f8a8f5be3720 consensus state has no leader
I20250112 20:51:35.780974 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_05c8c0dc76884c47b0e87e2801dffe72" } new_table_name: "default.Table_3fc149714fca4476ae1e1091f86a1fcc" modify_external_catalogs: true
I20250112 20:51:35.780236 1178 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.785198 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e9dd5f9cbc524e449863f6c9b52424b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.785856 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9dd5f9cbc524e449863f6c9b52424b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.786187 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_df6048c1d029498b8d19b7645fa5afd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.786667 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df6048c1d029498b8d19b7645fa5afd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.788925 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.790904 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9c55b922c39545ac9c89115cd587ea8c
I20250112 20:51:35.791332 453 catalog_manager.cc:6596] ReplaceTablet: tablet 9c55b922c39545ac9c89115cd587ea8c deleted and replaced by tablet be6751397eab403a8998a6a71820a6f9
I20250112 20:51:35.791712 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.789975 749 tablet_service.cc:1467] Processing CreateTablet for tablet 720aa779aada46d5a888d5011804a6a8 (DEFAULT_TABLE table=default.Table_eb174dc18b7d43d69147c2d69b5f458d [id=ff62e1f5b1aa45408d49878b0c050510]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.794341 1178 tablet_bootstrap.cc:654] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.801241 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3994d41a7ecd418a98a3c203add0d4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.813231 1178 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.813386 1178 ts_tablet_manager.cc:1397] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.002s
I20250112 20:51:35.814200 1178 raft_consensus.cc:357] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.814412 1178 raft_consensus.cc:383] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.814481 1178 raft_consensus.cc:738] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.814702 1178 consensus_queue.cc:260] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.818590 1186 ts_tablet_manager.cc:1918] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.826985 1186 log.cc:1198] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3c14102aecb743e2a82660039bd0f038
I20250112 20:51:35.827764 1186 ts_tablet_manager.cc:1939] T 3c14102aecb743e2a82660039bd0f038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.829066 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 455) took 1032 ms. Trace:
I20250112 20:51:35.830021 1186 rpcz_store.cc:276] 0112 20:51:34.796604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.796861 (+ 257us) service_pool.cc:225] Handling call
0112 20:51:35.757507 (+960646us) ts_tablet_manager.cc:1145] Deleting tablet 3c14102aecb743e2a82660039bd0f038
0112 20:51:35.757517 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.797368 (+ 39851us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.818484 (+ 21116us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.829049 (+ 10565us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":207,"tablet-delete.queue_time_us":960480}
I20250112 20:51:35.824715 1178 ts_tablet_manager.cc:1428] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 20:51:35.832384 1178 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.833312 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 (DEFAULT_TABLE table=default.Table_455466a0838742f3a2cc35cdab729433 [id=fada45a1d9a64ff4871c139b8767f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.834268 1178 tablet_bootstrap.cc:654] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.842058 1178 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.842185 1178 ts_tablet_manager.cc:1397] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250112 20:51:35.843080 1186 tablet_replica.cc:331] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.843138 1178 raft_consensus.cc:357] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.843317 1178 raft_consensus.cc:383] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.843376 1178 raft_consensus.cc:738] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.843604 1178 consensus_queue.cc:260] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.844082 1178 ts_tablet_manager.cc:1428] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:35.844420 1178 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:35.843248 1186 raft_consensus.cc:2238] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.847524 1186 raft_consensus.cc:2267] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.848050 1186 ts_tablet_manager.cc:1905] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.851089 1184 ts_tablet_manager.cc:1918] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.851248 1184 log.cc:1198] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3c14102aecb743e2a82660039bd0f038
I20250112 20:51:35.851615 1184 ts_tablet_manager.cc:1939] T 3c14102aecb743e2a82660039bd0f038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.852399 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 425) took 1460 ms. Trace:
I20250112 20:51:35.852511 1184 rpcz_store.cc:276] 0112 20:51:34.391621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.394782 (+ 3161us) service_pool.cc:225] Handling call
0112 20:51:35.614032 (+1219250us) ts_tablet_manager.cc:1145] Deleting tablet 3c14102aecb743e2a82660039bd0f038
0112 20:51:35.614039 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.639217 (+ 25178us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.850952 (+211735us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.852364 (+ 1412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196607}
I20250112 20:51:35.854934 1184 tablet_replica.cc:331] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.856168 1178 tablet_bootstrap.cc:654] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.857400 1942 consensus_queue.cc:1035] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.858342 1184 raft_consensus.cc:2238] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.858549 1184 raft_consensus.cc:2267] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.861147 1184 ts_tablet_manager.cc:1905] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.868353 1163 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.868573 1163 ts_tablet_manager.cc:1397] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.276s user 0.017s sys 0.028s
I20250112 20:51:35.868640 2040 tablet.cc:1716] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Skipping requested alter to schema version 2, tablet already version 2
I20250112 20:51:35.869464 1163 raft_consensus.cc:357] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.873839 1163 raft_consensus.cc:383] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.873952 1163 raft_consensus.cc:738] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.875936 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868ce9397b884b0095dae39ea57f5fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.877044 1163 consensus_queue.cc:260] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.877837 1163 ts_tablet_manager.cc:1428] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250112 20:51:35.878159 1163 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.881078 1163 tablet_bootstrap.cc:654] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:35.885442 1186 ts_tablet_manager.cc:1918] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:35.885881 1186 log.cc:1198] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8aaa5246382745ecbea4b9bd4825e8f1
I20250112 20:51:35.886480 1186 ts_tablet_manager.cc:1939] T 8aaa5246382745ecbea4b9bd4825e8f1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.887460 1178 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:35.887609 1178 ts_tablet_manager.cc:1397] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20250112 20:51:35.888396 1178 raft_consensus.cc:357] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.888619 1178 raft_consensus.cc:383] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.888686 1178 raft_consensus.cc:738] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.888928 1178 consensus_queue.cc:260] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:35.890023 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 (table default.Table_11a4b78b73c54798b358bc7647928f4e [id=d518a4c7edde4ba3b05d700c1cdb1dd1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b30a34c1abcb4f72a7f4f8a8f5be3720 consensus state has no leader
I20250112 20:51:35.890314 1186 tablet_replica.cc:331] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.890661 1931 raft_consensus.cc:491] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.890724 1186 raft_consensus.cc:2238] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.891301 2029 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.890908 2052 raft_consensus.cc:491] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.890852 1931 raft_consensus.cc:513] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.891577 2052 raft_consensus.cc:513] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:35.892206 1931 leader_election.cc:290] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.892352 1931 raft_consensus.cc:491] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.892379 2052 leader_election.cc:290] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:35.892598 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b84624dc685b401192ca753ff59da016 (table default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]) successfully deleted
I20250112 20:51:35.892788 2028 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.901188 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f249d70b116443b7823e093efeda5cc3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.901477 1047 raft_consensus.cc:2388] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:35.901650 1184 ts_tablet_manager.cc:1918] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.901788 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f0875e271d4aaf9dc5da6aa8113ed0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.901782 1184 log.cc:1198] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/70c86ad5f59d431e9b1a1fa890fd35c0
I20250112 20:51:35.903543 1184 ts_tablet_manager.cc:1939] T 70c86ad5f59d431e9b1a1fa890fd35c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:35.900983 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 101500us
I20250112 20:51:35.905699 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 427) took 1476 ms. Trace:
I20250112 20:51:35.905720 1931 raft_consensus.cc:513] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.906092 1184 rpcz_store.cc:276] 0112 20:51:34.429326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.429741 (+ 415us) service_pool.cc:225] Handling call
0112 20:51:35.854862 (+1425121us) ts_tablet_manager.cc:1145] Deleting tablet 70c86ad5f59d431e9b1a1fa890fd35c0
0112 20:51:35.854872 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.887531 (+ 32659us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.901545 (+ 14014us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.905333 (+ 3788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419576}
I20250112 20:51:35.907016 1184 tablet_replica.cc:331] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.907773 1184 raft_consensus.cc:2238] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.908298 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3a2ee62fff7d4cf7b4aaeacf63ed8e02 (table default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]) successfully deleted
I20250112 20:51:35.908514 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3c14102aecb743e2a82660039bd0f038 (table default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]) successfully deleted
I20250112 20:51:35.908886 1184 raft_consensus.cc:2267] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.909579 1186 raft_consensus.cc:2267] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.909699 1184 ts_tablet_manager.cc:1905] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.910121 1186 ts_tablet_manager.cc:1905] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.913882 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.914094 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.914348 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.914314 885 tablet_service.cc:1467] Processing CreateTablet for tablet 05550152671e4e28829b5b102b59f715 (DEFAULT_TABLE table=default.Table_c06477d484e0483589b0530cf5787b9c [id=4b364d74bdc149a09a4765f2c125d35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.914747 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05550152671e4e28829b5b102b59f715. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.914783 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4283dadebf89422795d890e3991db698" } modify_external_catalogs: true
I20250112 20:51:35.915022 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d8330fa1e4654a1a9f167a1120a103fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.915426 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.915701 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.915822 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c55b922c39545ac9c89115cd587ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet be6751397eab403a8998a6a71820a6f9) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:35.916152 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 05550152671e4e28829b5b102b59f715 (DEFAULT_TABLE table=default.Table_c06477d484e0483589b0530cf5787b9c [id=4b364d74bdc149a09a4765f2c125d35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:35.916564 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05550152671e4e28829b5b102b59f715. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:35.918597 1821 raft_consensus.cc:491] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:35.918731 1821 raft_consensus.cc:513] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:35.919142 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.919332 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c55b922c39545ac9c89115cd587ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet be6751397eab403a8998a6a71820a6f9) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.919425 1821 leader_election.cc:290] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.920568 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4283dadebf89422795d890e3991db698" } modify_external_catalogs: true
W20250112 20:51:35.920868 983 leader_election.cc:343] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:35.921262 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cea4e75bb1d34fcdb209c15b1cd6738d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.921983 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1d97ac93467a4a7a8ae44d52719f191c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d97ac93467a4a7a8ae44d52719f191c already in progress: creating tablet
I20250112 20:51:35.922334 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720aa779aada46d5a888d5011804a6a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:35.922680 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea4e75bb1d34fcdb209c15b1cd6738d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.923700 1185 tablet_replica.cc:331] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:35.924827 710 leader_election.cc:343] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:35.925465 710 leader_election.cc:343] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:35.925596 710 leader_election.cc:304] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:35.926755 710 leader_election.cc:343] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: f249d70b116443b7823e093efeda5cc3
I20250112 20:51:35.926896 710 leader_election.cc:304] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:35.928580 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c13126e53b46c0a1e7beafe1b0d295" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:35.928961 1046 raft_consensus.cc:2388] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:35.929988 1185 raft_consensus.cc:2238] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.931649 1931 leader_election.cc:290] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:35.932677 738 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.932902 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c55b922c39545ac9c89115cd587ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet be6751397eab403a8998a6a71820a6f9) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:35.934459 2052 raft_consensus.cc:3054] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:35.935848 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1d60604713524dba8f3a42c4f3bc7300 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d60604713524dba8f3a42c4f3bc7300 already in progress: creating tablet
I20250112 20:51:35.936767 1163 tablet_bootstrap.cc:492] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.936939 1163 ts_tablet_manager.cc:1397] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.006s
I20250112 20:51:35.940176 1163 raft_consensus.cc:357] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:35.940310 770 raft_consensus.cc:3054] T bc4629339765453d988953799095b3ac P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.940416 1163 raft_consensus.cc:383] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.940495 1163 raft_consensus.cc:738] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.941183 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:35.942620 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9570269368e14b1abeaffdcc741c6f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9570269368e14b1abeaffdcc741c6f3a already in progress: creating tablet
I20250112 20:51:35.942755 1186 ts_tablet_manager.cc:1918] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:35.942858 1186 log.cc:1198] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:35.943117 1186 ts_tablet_manager.cc:1939] T 6fe28f23448c47d3979f1982f939e39c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:35.943598 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d17e6d263014448b96c5d1f03013e6e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17e6d263014448b96c5d1f03013e6e7 already in progress: creating tablet
I20250112 20:51:35.944968 1163 consensus_queue.cc:260] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:35.945279 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d7117edf078848c0a1c83ab8e804dfed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7117edf078848c0a1c83ab8e804dfed already in progress: creating tablet
W20250112 20:51:35.945737 710 leader_election.cc:343] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 83c13126e53b46c0a1e7beafe1b0d295
I20250112 20:51:35.946074 1164 tablet_bootstrap.cc:654] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:35.945654 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 already in progress: creating tablet
W20250112 20:51:35.946842 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d770e0399e80491f8de9b08d1eb2f139 because the tablet was not found. No further retry: Not found: Tablet not found: d770e0399e80491f8de9b08d1eb2f139
I20250112 20:51:35.947016 2059 raft_consensus.cc:2701] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250112 20:51:35.948161 1185 raft_consensus.cc:2267] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.948676 710 leader_election.cc:304] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:35.949039 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f249d70b116443b7823e093efeda5cc3 because the tablet was not found. No further retry: Not found: Tablet not found: f249d70b116443b7823e093efeda5cc3
I20250112 20:51:35.951320 1184 ts_tablet_manager.cc:1918] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.951486 1184 log.cc:1198] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f769533bfde0448792d01b883e41763c
W20250112 20:51:35.952250 983 leader_election.cc:343] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:35.952378 983 leader_election.cc:304] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:35.952867 1184 ts_tablet_manager.cc:1939] T f769533bfde0448792d01b883e41763c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:35.953250 1186 tablet_replica.cc:331] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:35.953441 1186 raft_consensus.cc:2238] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:35.953455 1821 raft_consensus.cc:2743] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.954126 1186 pending_rounds.cc:62] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:35.954201 1163 ts_tablet_manager.cc:1428] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 20:51:35.954190 1186 pending_rounds.cc:69] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585031422337024 op_type: NO_OP noop_request { }
I20250112 20:51:35.954308 1186 raft_consensus.cc:2883] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:35.954396 1186 raft_consensus.cc:2267] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:35.949265 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e590f9615c4d47abaad1a36ac74b3c80 because the tablet was not found. No further retry: Not found: Tablet not found: e590f9615c4d47abaad1a36ac74b3c80
W20250112 20:51:35.955574 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8c38c8594a384689ae57c26174765284 because the tablet was not found. No further retry: Not found: Tablet not found: 8c38c8594a384689ae57c26174765284
I20250112 20:51:35.956224 2040 tablet.cc:1721] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:35.957347 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 424) took 1565 ms. Trace:
I20250112 20:51:35.957512 1184 rpcz_store.cc:276] 0112 20:51:34.391501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.396291 (+ 4790us) service_pool.cc:225] Handling call
0112 20:51:35.906935 (+1510644us) ts_tablet_manager.cc:1145] Deleting tablet f769533bfde0448792d01b883e41763c
0112 20:51:35.906945 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.919037 (+ 12092us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.951168 (+ 32131us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:35.957303 (+ 6135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465341}
I20250112 20:51:35.958035 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 428) took 1491 ms. Trace:
I20250112 20:51:35.958166 1185 ts_tablet_manager.cc:1905] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.958174 1184 rpcz_store.cc:276] 0112 20:51:34.466816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.467834 (+ 1018us) service_pool.cc:225] Handling call
0112 20:51:35.957848 (+1490014us) ts_tablet_manager.cc:1145] Deleting tablet f0daaeb0bf6046da8de969b1f4c71c84
0112 20:51:35.957859 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.957988 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482496}
I20250112 20:51:35.958585 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 429) took 1469 ms. Trace:
I20250112 20:51:35.958712 1184 rpcz_store.cc:276] 0112 20:51:34.489222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.489377 (+ 155us) service_pool.cc:225] Handling call
0112 20:51:35.958433 (+1469056us) ts_tablet_manager.cc:1145] Deleting tablet f3ff8ac6f4d94c10b03a764bbf910c44
0112 20:51:35.958444 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.958539 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468850}
I20250112 20:51:35.959007 1184 tablet_replica.cc:331] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:35.959193 1184 raft_consensus.cc:2238] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:35.959398 1184 raft_consensus.cc:2267] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:35.959957 1931 raft_consensus.cc:3054] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:35.960088 1184 ts_tablet_manager.cc:1905] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.961035 1163 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:35.962742 1982 consensus_queue.cc:1035] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:35.963404 1163 tablet_bootstrap.cc:654] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:35.966094 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 70c86ad5f59d431e9b1a1fa890fd35c0 because the tablet was not found. No further retry: Not found: Tablet not found: 70c86ad5f59d431e9b1a1fa890fd35c0
I20250112 20:51:35.966337 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_15e3fd8e71984a81acb2dabf8f3563f7 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]) successfully deleted
I20250112 20:51:35.966706 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 6fe28f23448c47d3979f1982f939e39c (table default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]) successfully deleted
W20250112 20:51:35.966912 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e72fda1743fc4f96b29add6e36e884de because the tablet was not found. No further retry: Not found: Tablet not found: e72fda1743fc4f96b29add6e36e884de
W20250112 20:51:35.967114 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9880e7d303b54d959bca4e8464a6b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9880e7d303b54d959bca4e8464a6b30c already in progress: creating tablet
I20250112 20:51:35.969373 1931 raft_consensus.cc:2743] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:35.970582 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efa7f6a1e9804e6ab822f9d4c004b1b3 already in progress: creating tablet
W20250112 20:51:35.970855 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5d167065029a46b791d413db8a2c4bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d167065029a46b791d413db8a2c4bcc already in progress: creating tablet
W20250112 20:51:35.971098 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 921ae136d5e2438e950b96f156f23da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 921ae136d5e2438e950b96f156f23da3 already in progress: creating tablet
W20250112 20:51:35.971320 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ef01b41b6140d6bdad07bc79dd7dc5 already in progress: creating tablet
W20250112 20:51:35.971496 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 267ebb3c30194d9fa6a092d9bec4eadf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267ebb3c30194d9fa6a092d9bec4eadf already in progress: creating tablet
I20250112 20:51:35.972016 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:35.972484 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.973174 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.973204 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3c14102aecb743e2a82660039bd0f038 (table default.Table_e18e998a23124368951dc9ad456b0567 [id=f0f36f19644b466fbe7cc5e57b0a9075]) successfully deleted
I20250112 20:51:35.973574 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 70c86ad5f59d431e9b1a1fa890fd35c0 (table default.Table_97ae975748324aa3a68452cc348b9e95 [id=f62aab576697492ab994a1b35fd52c55]) successfully deleted
I20250112 20:51:35.973937 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f769533bfde0448792d01b883e41763c (table default.Table_0218ef4723ae4dcd96e64a39634592cf [id=956592813ab34ba2a7633563381a4788]) successfully deleted
W20250112 20:51:35.974138 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f0daaeb0bf6046da8de969b1f4c71c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0daaeb0bf6046da8de969b1f4c71c84 already in progress: creating tablet
W20250112 20:51:35.974390 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f3ff8ac6f4d94c10b03a764bbf910c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3ff8ac6f4d94c10b03a764bbf910c44 already in progress: creating tablet
I20250112 20:51:35.974339 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d184983b685e4a4cb571a72c242ef9b3" } new_table_name: "default.Table_c04acf4e49804989ad6fbb05c89a8f20" modify_external_catalogs: true
I20250112 20:51:35.974594 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6b593fd0b27949e2b15bfc7da5a989be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:35.975077 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b593fd0b27949e2b15bfc7da5a989be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:35.975229 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7deb6e37f75e4aebacb04c4553049940 with delete_type TABLET_DATA_DELETED (replaced by tablet 669e1cfd407e4d87a63e6832d9c6fa1f) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:35.975773 2052 raft_consensus.cc:2743] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:35.977864 1186 ts_tablet_manager.cc:1905] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:35.993865 1163 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:35.994444 1186 ts_tablet_manager.cc:1918] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:35.995018 1163 ts_tablet_manager.cc:1397] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250112 20:51:35.995849 1186 log.cc:1198] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8f636df32bd34438bb54989736eaa431
I20250112 20:51:35.996026 1163 raft_consensus.cc:357] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.996223 1163 raft_consensus.cc:383] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:35.996313 1163 raft_consensus.cc:738] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:35.996552 1163 consensus_queue.cc:260] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:35.996961 1186 ts_tablet_manager.cc:1939] T 8f636df32bd34438bb54989736eaa431 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:35.998579 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8f636df32bd34438bb54989736eaa431 (table default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]) successfully deleted
I20250112 20:51:35.999832 1163 ts_tablet_manager.cc:1428] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:36.000182 1163 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.002699 1186 tablet_replica.cc:331] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.004148 1821 consensus_queue.cc:1035] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:36.005231 1163 tablet_bootstrap.cc:654] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.006944 1186 raft_consensus.cc:2238] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.007119 1186 raft_consensus.cc:2267] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.008319 1186 ts_tablet_manager.cc:1905] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.010766 1185 ts_tablet_manager.cc:1918] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.010958 1185 log.cc:1198] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:36.011395 1185 ts_tablet_manager.cc:1939] T 81f0875e271d4aaf9dc5da6aa8113ed0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.012403 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
W20250112 20:51:36.014331 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8aaa5246382745ecbea4b9bd4825e8f1 because the tablet was not found. No further retry: Not found: Tablet not found: 8aaa5246382745ecbea4b9bd4825e8f1
I20250112 20:51:36.017825 1185 tablet_replica.cc:331] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.018044 1185 raft_consensus.cc:2238] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.018210 1185 raft_consensus.cc:2267] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.024379 1942 raft_consensus.cc:491] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.024544 1942 raft_consensus.cc:513] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.025466 1942 leader_election.cc:290] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.026376 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eb0f79df5444f8a8360921d5f2b9ee" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.026453 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eb0f79df5444f8a8360921d5f2b9ee" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:36.027041 847 leader_election.cc:343] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:36.027549 847 leader_election.cc:343] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.027606 1185 ts_tablet_manager.cc:1905] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.027662 847 leader_election.cc:304] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.028203 1942 raft_consensus.cc:2743] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.033753 1178 ts_tablet_manager.cc:1428] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.146s user 0.002s sys 0.000s
I20250112 20:51:36.034099 1178 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.033872 1186 ts_tablet_manager.cc:1918] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.036403 1186 log.cc:1198] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d8330fa1e4654a1a9f167a1120a103fc
I20250112 20:51:36.036712 1186 ts_tablet_manager.cc:1939] T d8330fa1e4654a1a9f167a1120a103fc P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.038276 1824 consensus_queue.cc:1035] T bc4629339765453d988953799095b3ac P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:36.039322 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d8330fa1e4654a1a9f167a1120a103fc (table default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]) successfully deleted
I20250112 20:51:36.041499 1186 tablet_replica.cc:331] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.041687 1186 raft_consensus.cc:2238] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.041819 1186 raft_consensus.cc:2267] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.042336 1186 ts_tablet_manager.cc:1905] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:36.049358 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db9bab2477a94c169817d9de8aaad1c0 consensus state has no leader
I20250112 20:51:36.049618 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f4cae067df1a40098f81cfbd17224c75 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.050261 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e907fd649a434fe9b3a51b2619964cb3" } modify_external_catalogs: true
I20250112 20:51:36.050432 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e907fd649a434fe9b3a51b2619964cb3" } modify_external_catalogs: true
I20250112 20:51:36.051205 1184 ts_tablet_manager.cc:1918] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:36.051451 1184 log.cc:1198] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/decdec0d34344ec4b422643b0d4e7659
I20250112 20:51:36.052048 1184 ts_tablet_manager.cc:1939] T decdec0d34344ec4b422643b0d4e7659 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.054360 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 430) took 1564 ms. Trace:
I20250112 20:51:36.055289 1163 tablet_bootstrap.cc:492] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.055533 1163 ts_tablet_manager.cc:1397] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.000s
I20250112 20:51:36.056342 1178 tablet_bootstrap.cc:654] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.056335 1163 raft_consensus.cc:357] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:36.056665 1163 raft_consensus.cc:383] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.056823 1163 raft_consensus.cc:738] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.058552 1164 tablet_bootstrap.cc:492] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.058718 1164 ts_tablet_manager.cc:1397] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.332s user 0.002s sys 0.011s
I20250112 20:51:36.059504 1164 raft_consensus.cc:357] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.059752 1164 raft_consensus.cc:383] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.059825 1164 raft_consensus.cc:738] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:36.060286 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 consensus state has no leader
I20250112 20:51:36.060827 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_bf1f1d5c2be64a72af1c3c8396cb7180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.061233 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_893cdd72c53441f3b51cd4eeb6cda9f6" } modify_external_catalogs: true
I20250112 20:51:36.061415 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_893cdd72c53441f3b51cd4eeb6cda9f6" } modify_external_catalogs: true
I20250112 20:51:36.061784 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.062835 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fc6da3983cb406e8e10ecf3f3fd239b" } modify_external_catalogs: true
I20250112 20:51:36.062983 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fc6da3983cb406e8e10ecf3f3fd239b" } modify_external_catalogs: true
I20250112 20:51:36.060073 1164 consensus_queue.cc:260] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.061424 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf1f1d5c2be64a72af1c3c8396cb7180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.067088 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2de03e3114d145aeb81590dbae2c548e (DEFAULT_TABLE table=default.Table_0ede004124b94194a25701ee39ebfc60 [id=e221237342294f5884958606d2bf6ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.067199 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:36.067478 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de03e3114d145aeb81590dbae2c548e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.067464 449 catalog_manager.cc:5526] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:36.070608 1164 ts_tablet_manager.cc:1428] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:36.071429 1163 consensus_queue.cc:260] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:36.073942 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 2de03e3114d145aeb81590dbae2c548e (DEFAULT_TABLE table=default.Table_0ede004124b94194a25701ee39ebfc60 [id=e221237342294f5884958606d2bf6ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.074159 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d (DEFAULT_TABLE table=default.Table_06d8dc6fe3af4be68fee565a90654657 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.074412 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de03e3114d145aeb81590dbae2c548e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.074410 1017 tablet_service.cc:1467] Processing CreateTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d (DEFAULT_TABLE table=default.Table_a28d1068edaf4014a962ab336f9243d5 [id=9bdfb314140b4de299f62c9c21ecdf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.074716 1012 tablet_service.cc:1467] Processing CreateTablet for tablet f8d521e6c8cd461887f93f395c27b89e (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.074994 1164 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.075258 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 913149098822406e8d4ca5a440c9afd7 (DEFAULT_TABLE table=default.Table_341aa7fe62034251879c0e39f76dd6e0 [id=46618cb6d366443bb29b26bc8e43a2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.075762 1011 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 (DEFAULT_TABLE table=default.Table_1f59b364b3b3490aaa782d7e83f048ff [id=ccd2497cb68442aca96b031bf9041afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.075434 1184 rpcz_store.cc:276] 0112 20:51:34.489759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.489799 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:35.958950 (+1469151us) ts_tablet_manager.cc:1145] Deleting tablet decdec0d34344ec4b422643b0d4e7659
0112 20:51:35.958957 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:35.991899 (+ 32942us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.051040 (+ 59141us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.054306 (+ 3266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468992}
I20250112 20:51:36.074748 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 015f54bdae6b4333963f9e085c5189e8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.076277 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet decdec0d34344ec4b422643b0d4e7659 (table default.Table_3fc149714fca4476ae1e1091f86a1fcc [id=2f68c758f6554102961912265cf48921]) successfully deleted
I20250112 20:51:36.076566 1821 raft_consensus.cc:491] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.074781 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37e9e95c6b745ccaba5446b04abcc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.076716 1821 raft_consensus.cc:513] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.077018 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.077585 1821 leader_election.cc:290] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.077840 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb6a5146c44edea076a10da58ea1a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:36.078719 983 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:36.081148 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d521e6c8cd461887f93f395c27b89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.085080 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913149098822406e8d4ca5a440c9afd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:36.087152 983 leader_election.cc:343] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:36.087325 983 leader_election.cc:304] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.087832 1821 raft_consensus.cc:2743] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.088891 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f54bdae6b4333963f9e085c5189e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.093791 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2de03e3114d145aeb81590dbae2c548e (DEFAULT_TABLE table=default.Table_0ede004124b94194a25701ee39ebfc60 [id=e221237342294f5884958606d2bf6ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.094233 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de03e3114d145aeb81590dbae2c548e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.094377 752 tablet_service.cc:1467] Processing CreateTablet for tablet f8d521e6c8cd461887f93f395c27b89e (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.094694 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d521e6c8cd461887f93f395c27b89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.096530 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 437) took 1433 ms. Trace:
I20250112 20:51:36.096621 1184 rpcz_store.cc:276] 0112 20:51:34.663476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.667949 (+ 4473us) service_pool.cc:225] Handling call
0112 20:51:36.096369 (+1428420us) ts_tablet_manager.cc:1145] Deleting tablet 61b74aaf7f5b4c87acbefaa4df5c10d2
0112 20:51:36.096378 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.096480 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428207}
I20250112 20:51:36.097054 756 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 (DEFAULT_TABLE table=default.Table_1f59b364b3b3490aaa782d7e83f048ff [id=ccd2497cb68442aca96b031bf9041afd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:36.097100 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 already in progress: creating tablet
I20250112 20:51:36.097272 753 tablet_service.cc:1467] Processing CreateTablet for tablet 913149098822406e8d4ca5a440c9afd7 (DEFAULT_TABLE table=default.Table_341aa7fe62034251879c0e39f76dd6e0 [id=46618cb6d366443bb29b26bc8e43a2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.097406 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb6a5146c44edea076a10da58ea1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.097510 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 438) took 1382 ms. Trace:
I20250112 20:51:36.097582 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913149098822406e8d4ca5a440c9afd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.097597 1184 rpcz_store.cc:276] 0112 20:51:34.715081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.715362 (+ 281us) service_pool.cc:225] Handling call
0112 20:51:36.097359 (+1381997us) ts_tablet_manager.cc:1145] Deleting tablet 70653110aec74727acec9d63c2eb0f05
0112 20:51:36.097368 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.097464 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381714}
I20250112 20:51:36.097048 739 tablet_service.cc:1467] Processing CreateTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d (DEFAULT_TABLE table=default.Table_06d8dc6fe3af4be68fee565a90654657 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:36.097977 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 70653110aec74727acec9d63c2eb0f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70653110aec74727acec9d63c2eb0f05 already in progress: creating tablet
I20250112 20:51:36.098079 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.098336 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 439) took 1366 ms. Trace:
I20250112 20:51:36.098420 1184 rpcz_store.cc:276] 0112 20:51:34.731438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.731503 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:36.098202 (+1366699us) ts_tablet_manager.cc:1145] Deleting tablet a3ca4d49496e42ddb263264ceef2f27a
0112 20:51:36.098211 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.098291 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366408}
W20250112 20:51:36.098789 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a3ca4d49496e42ddb263264ceef2f27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ca4d49496e42ddb263264ceef2f27a already in progress: creating tablet
I20250112 20:51:36.099363 1184 tablet_replica.cc:331] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.099597 1184 raft_consensus.cc:2238] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.099792 1184 raft_consensus.cc:2267] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.100176 738 tablet_service.cc:1467] Processing CreateTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d (DEFAULT_TABLE table=default.Table_a28d1068edaf4014a962ab336f9243d5 [id=9bdfb314140b4de299f62c9c21ecdf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.101094 885 tablet_service.cc:1467] Processing CreateTablet for tablet f8d521e6c8cd461887f93f395c27b89e (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.101552 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d521e6c8cd461887f93f395c27b89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.101610 744 tablet_service.cc:1467] Processing CreateTablet for tablet 015f54bdae6b4333963f9e085c5189e8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.101714 886 tablet_service.cc:1467] Processing CreateTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d (DEFAULT_TABLE table=default.Table_a28d1068edaf4014a962ab336f9243d5 [id=9bdfb314140b4de299f62c9c21ecdf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.102039 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37e9e95c6b745ccaba5446b04abcc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.102113 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f54bdae6b4333963f9e085c5189e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.103037 874 tablet_service.cc:1467] Processing CreateTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d (DEFAULT_TABLE table=default.Table_06d8dc6fe3af4be68fee565a90654657 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.103322 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.104831 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37e9e95c6b745ccaba5446b04abcc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.105055 880 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 (DEFAULT_TABLE table=default.Table_1f59b364b3b3490aaa782d7e83f048ff [id=ccd2497cb68442aca96b031bf9041afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.105279 1184 ts_tablet_manager.cc:1905] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.105350 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb6a5146c44edea076a10da58ea1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.108929 882 tablet_service.cc:1467] Processing CreateTablet for tablet 913149098822406e8d4ca5a440c9afd7 (DEFAULT_TABLE table=default.Table_341aa7fe62034251879c0e39f76dd6e0 [id=46618cb6d366443bb29b26bc8e43a2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.109022 884 tablet_service.cc:1467] Processing CreateTablet for tablet 015f54bdae6b4333963f9e085c5189e8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.109258 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913149098822406e8d4ca5a440c9afd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.109503 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015f54bdae6b4333963f9e085c5189e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.110687 1821 raft_consensus.cc:491] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.110845 1821 raft_consensus.cc:513] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.111421 1821 leader_election.cc:290] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.112061 775 raft_consensus.cc:3054] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.112263 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:36.112706 983 leader_election.cc:343] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.114179 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:36.114614 983 leader_election.cc:343] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.114727 983 leader_election.cc:304] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.115123 1821 raft_consensus.cc:2743] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.119028 1745 raft_consensus.cc:491] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.119170 1745 raft_consensus.cc:513] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.120296 1745 leader_election.cc:290] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.122027 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4688dd6bbfc046b7bb140206cae9f104" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:36.122602 710 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.126250 1184 ts_tablet_manager.cc:1918] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.126399 1184 log.cc:1198] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e590f9615c4d47abaad1a36ac74b3c80
I20250112 20:51:36.126726 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4688dd6bbfc046b7bb140206cae9f104" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.126829 1184 ts_tablet_manager.cc:1939] T e590f9615c4d47abaad1a36ac74b3c80 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:36.127329 710 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:36.127596 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 440) took 1377 ms. Trace:
I20250112 20:51:36.127699 710 leader_election.cc:304] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.128008 1745 raft_consensus.cc:2743] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.139273 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9854d410371b442c81705698f427d8b0 (DEFAULT_TABLE table=default.Table_09c966930c7b4277a84b783534270688 [id=448df32415214bff8860564b9b8fa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.139698 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9854d410371b442c81705698f427d8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.141036 1184 rpcz_store.cc:276] 0112 20:51:34.750342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.750393 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:36.099272 (+1348879us) ts_tablet_manager.cc:1145] Deleting tablet e590f9615c4d47abaad1a36ac74b3c80
0112 20:51:36.099284 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.108292 (+ 9008us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.126116 (+ 17824us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.127557 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348668}
I20250112 20:51:36.141650 1184 tablet_replica.cc:331] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.141961 1184 raft_consensus.cc:2238] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.142304 1184 raft_consensus.cc:2267] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.143378 1184 ts_tablet_manager.cc:1905] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.145112 754 tablet_service.cc:1467] Processing CreateTablet for tablet 85683e97dae44399b1dc7bbd54d62eb6 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.145538 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85683e97dae44399b1dc7bbd54d62eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.149859 741 tablet_service.cc:1467] Processing CreateTablet for tablet f52a9216c5044f88a0060e3e8e95eddd (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.150256 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52a9216c5044f88a0060e3e8e95eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.153569 885 tablet_service.cc:1467] Processing CreateTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 (DEFAULT_TABLE table=default.Table_10962594cedf43f6a5533239b618afa4 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.153831 877 tablet_service.cc:1467] Processing CreateTablet for tablet f52a9216c5044f88a0060e3e8e95eddd (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.153942 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60eefdbabc174cd6a67fd12ea7d95071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.154141 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52a9216c5044f88a0060e3e8e95eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.154474 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9854d410371b442c81705698f427d8b0 (DEFAULT_TABLE table=default.Table_09c966930c7b4277a84b783534270688 [id=448df32415214bff8860564b9b8fa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.154757 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9854d410371b442c81705698f427d8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.157068 874 tablet_service.cc:1467] Processing CreateTablet for tablet 85683e97dae44399b1dc7bbd54d62eb6 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.157356 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85683e97dae44399b1dc7bbd54d62eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.159828 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e590f9615c4d47abaad1a36ac74b3c80 (table default.Table_4c02bae8e4d64469b992b13dd36c7e81 [id=7dc6e9f691804365a9696db3b26700bd]) successfully deleted
W20250112 20:51:36.163805 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39271, user_credentials={real_user=slave}} blocked reactor thread for 107188us
I20250112 20:51:36.171023 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 (DEFAULT_TABLE table=default.Table_10962594cedf43f6a5533239b618afa4 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.171464 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60eefdbabc174cd6a67fd12ea7d95071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.175971 1164 tablet_bootstrap.cc:654] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.179103 1185 ts_tablet_manager.cc:1918] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.179284 1185 log.cc:1198] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8f636df32bd34438bb54989736eaa431
I20250112 20:51:36.179486 1184 ts_tablet_manager.cc:1918] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.179644 1185 ts_tablet_manager.cc:1939] T 8f636df32bd34438bb54989736eaa431 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.179565 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 85683e97dae44399b1dc7bbd54d62eb6 (DEFAULT_TABLE table=default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.179811 1184 log.cc:1198] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/64cd83b199bb4da28d39da449945c389
I20250112 20:51:36.180024 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85683e97dae44399b1dc7bbd54d62eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.184381 1942 raft_consensus.cc:491] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.184517 1942 raft_consensus.cc:513] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.185115 1942 leader_election.cc:290] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.185674 747 tablet_service.cc:1467] Processing CreateTablet for tablet e2c41c2c92b442b986d6b75708c7420c (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.187423 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c41c2c92b442b986d6b75708c7420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.187363 739 tablet_service.cc:1467] Processing CreateTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 (DEFAULT_TABLE table=default.Table_10962594cedf43f6a5533239b618afa4 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.187767 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60eefdbabc174cd6a67fd12ea7d95071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.180306 1184 ts_tablet_manager.cc:1939] T 64cd83b199bb4da28d39da449945c389 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.192903 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 441) took 1405 ms. Trace:
I20250112 20:51:36.193430 1184 rpcz_store.cc:276] 0112 20:51:34.787300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.787766 (+ 466us) service_pool.cc:225] Handling call
0112 20:51:36.141567 (+1353801us) ts_tablet_manager.cc:1145] Deleting tablet 64cd83b199bb4da28d39da449945c389
0112 20:51:36.141582 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.147449 (+ 5867us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.179375 (+ 31926us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.192849 (+ 13474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1353587}
I20250112 20:51:36.194279 1184 tablet_replica.cc:331] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.194526 1184 raft_consensus.cc:2238] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.194767 1184 raft_consensus.cc:2267] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.195355 1184 ts_tablet_manager.cc:1905] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.185189 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e2c41c2c92b442b986d6b75708c7420c (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.195768 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c41c2c92b442b986d6b75708c7420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.198051 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 61964ec077094c018c5a03ad7699c9c6 (DEFAULT_TABLE table=default.Table_743de41614e248bd9cffca489bc49570 [id=e8f7a569b2914e54b0819c8387667edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.198408 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.198473 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61964ec077094c018c5a03ad7699c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.198798 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.198727 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.198901 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 64cd83b199bb4da28d39da449945c389 (table default.Table_0afc27e58eec42daadf528a66340c624 [id=370444777ab4411f837c19bc3824376a]) successfully deleted
I20250112 20:51:36.199286 1022 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 (DEFAULT_TABLE table=default.Table_0b49521c49204a03b257a7b38a4132c2 [id=08d133fd345241bab23273e839ceda6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.199666 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6d73ab981c4e87aa229a49fb3b6913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.199662 1011 tablet_service.cc:1467] Processing CreateTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 (DEFAULT_TABLE table=default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:36.200171 847 leader_election.cc:343] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.200233 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b722b0eb5944a4b344a3833905a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.200578 874 tablet_service.cc:1467] Processing CreateTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 (DEFAULT_TABLE table=default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.200933 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b722b0eb5944a4b344a3833905a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.189052 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f52a9216c5044f88a0060e3e8e95eddd (DEFAULT_TABLE table=default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.204712 880 tablet_service.cc:1467] Processing CreateTablet for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.205127 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.205185 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52a9216c5044f88a0060e3e8e95eddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.209030 886 tablet_service.cc:1467] Processing CreateTablet for tablet e2c41c2c92b442b986d6b75708c7420c (DEFAULT_TABLE table=default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.209426 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c41c2c92b442b986d6b75708c7420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.215355 1186 ts_tablet_manager.cc:1918] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.217850 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.221546 756 tablet_service.cc:1467] Processing CreateTablet for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 (DEFAULT_TABLE table=default.Table_343ab21a9b504bef9d7ceb9a38d3870c [id=537bf8b60c0249efb50df78260ed60c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.221951 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.222330 754 tablet_service.cc:1467] Processing CreateTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 (DEFAULT_TABLE table=default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.222676 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b722b0eb5944a4b344a3833905a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.225056 741 tablet_service.cc:1467] Processing CreateTablet for tablet 61964ec077094c018c5a03ad7699c9c6 (DEFAULT_TABLE table=default.Table_743de41614e248bd9cffca489bc49570 [id=e8f7a569b2914e54b0819c8387667edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.225425 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61964ec077094c018c5a03ad7699c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.225062 753 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 (DEFAULT_TABLE table=default.Table_0b49521c49204a03b257a7b38a4132c2 [id=08d133fd345241bab23273e839ceda6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.226099 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6d73ab981c4e87aa229a49fb3b6913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.075302 1163 ts_tablet_manager.cc:1428] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250112 20:51:36.232158 1163 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.234052 1163 tablet_bootstrap.cc:654] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.234643 1164 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.234828 1164 ts_tablet_manager.cc:1397] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.163s user 0.001s sys 0.004s
I20250112 20:51:36.180581 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9854d410371b442c81705698f427d8b0 (DEFAULT_TABLE table=default.Table_09c966930c7b4277a84b783534270688 [id=448df32415214bff8860564b9b8fa601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.235316 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9854d410371b442c81705698f427d8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.235671 1164 raft_consensus.cc:357] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.235883 1164 raft_consensus.cc:383] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.235978 1164 raft_consensus.cc:738] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.236621 1164 consensus_queue.cc:260] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.238262 1164 ts_tablet_manager.cc:1428] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:36.239185 1164 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.241542 1164 tablet_bootstrap.cc:654] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.255867 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1175f36a82364f40bf29fd45784c07a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.256484 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1175f36a82364f40bf29fd45784c07a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.256989 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]). Expected version 1 got 0
I20250112 20:51:36.256736 1164 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.257273 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:36.257390 1164 ts_tablet_manager.cc:1397] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250112 20:51:36.258184 1164 raft_consensus.cc:357] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.258404 1164 raft_consensus.cc:383] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.258466 1164 raft_consensus.cc:738] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.259075 1184 ts_tablet_manager.cc:1918] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.259604 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:36.261404 1164 consensus_queue.cc:260] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.263213 1164 ts_tablet_manager.cc:1428] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:36.264963 1184 log.cc:1198] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f249d70b116443b7823e093efeda5cc3
I20250112 20:51:36.265486 1184 ts_tablet_manager.cc:1939] T f249d70b116443b7823e093efeda5cc3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.266496 1164 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.268364 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 443) took 1480 ms. Trace:
I20250112 20:51:36.269019 1184 rpcz_store.cc:276] 0112 20:51:34.787565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.788016 (+ 451us) service_pool.cc:225] Handling call
0112 20:51:36.194046 (+1406030us) ts_tablet_manager.cc:1145] Deleting tablet f249d70b116443b7823e093efeda5cc3
0112 20:51:36.194204 (+ 158us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.227369 (+ 33165us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.258908 (+ 31539us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.268326 (+ 9418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405884}
I20250112 20:51:36.269556 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f249d70b116443b7823e093efeda5cc3 (table default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]) successfully deleted
I20250112 20:51:36.269961 1184 tablet_replica.cc:331] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.270232 1184 raft_consensus.cc:2238] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.270419 1184 raft_consensus.cc:2267] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.270951 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_0a377dabb6b14129ba1e8519d731bafb [id=11743480162f4442b00dc54a796e3461] alter complete (version 2)
I20250112 20:51:36.271049 1184 ts_tablet_manager.cc:1905] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.273960 1164 tablet_bootstrap.cc:654] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.274920 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f0875e271d4aaf9dc5da6aa8113ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9856e6b625d04f17b0869cee08f69142) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.275417 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.280807 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f249d70b116443b7823e093efeda5cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.283308 1164 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.283871 1164 ts_tablet_manager.cc:1397] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250112 20:51:36.289760 1184 ts_tablet_manager.cc:1918] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.284925 1164 raft_consensus.cc:357] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.292603 1164 raft_consensus.cc:383] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.292848 1164 raft_consensus.cc:738] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.292896 1184 log.cc:1198] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d8330fa1e4654a1a9f167a1120a103fc
I20250112 20:51:36.293300 1184 ts_tablet_manager.cc:1939] T d8330fa1e4654a1a9f167a1120a103fc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.293484 1164 consensus_queue.cc:260] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.294344 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 444) took 1506 ms. Trace:
I20250112 20:51:36.294481 1184 rpcz_store.cc:276] 0112 20:51:34.787674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.787758 (+ 84us) service_pool.cc:225] Handling call
0112 20:51:36.269876 (+1482118us) ts_tablet_manager.cc:1145] Deleting tablet d8330fa1e4654a1a9f167a1120a103fc
0112 20:51:36.269891 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.285861 (+ 15970us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.289645 (+ 3784us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.294291 (+ 4646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481653}
I20250112 20:51:36.294693 1164 ts_tablet_manager.cc:1428] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:36.295102 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d8330fa1e4654a1a9f167a1120a103fc (table default.Table_bc82df6f9ef0472db33f87ad8da54503 [id=9e548dcf18274d4a88ea50a0c6486700]) successfully deleted
I20250112 20:51:36.295485 1184 tablet_replica.cc:331] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.295615 1164 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.295755 1184 raft_consensus.cc:2238] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.298136 1164 tablet_bootstrap.cc:654] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.298535 1184 raft_consensus.cc:2267] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.299474 1184 ts_tablet_manager.cc:1905] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.304746 1164 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.308907 1164 ts_tablet_manager.cc:1397] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20250112 20:51:36.310982 1164 raft_consensus.cc:357] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.311596 1184 ts_tablet_manager.cc:1918] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.311847 1164 raft_consensus.cc:383] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.312255 1164 raft_consensus.cc:738] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.312590 1184 log.cc:1198] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/23d5269a5c8c4c049265e18bff4e2c45
I20250112 20:51:36.312629 1164 consensus_queue.cc:260] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.313088 1184 ts_tablet_manager.cc:1939] T 23d5269a5c8c4c049265e18bff4e2c45 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.313854 1164 ts_tablet_manager.cc:1428] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:36.314612 1164 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.314925 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 442) took 1527 ms. Trace:
I20250112 20:51:36.316242 1184 rpcz_store.cc:276] 0112 20:51:34.787437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:34.788286 (+ 849us) service_pool.cc:225] Handling call
0112 20:51:36.295397 (+1507111us) ts_tablet_manager.cc:1145] Deleting tablet 23d5269a5c8c4c049265e18bff4e2c45
0112 20:51:36.295407 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.306030 (+ 10623us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.311450 (+ 5420us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.314879 (+ 3429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506917}
I20250112 20:51:36.320732 1745 raft_consensus.cc:491] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.320885 1745 raft_consensus.cc:513] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.320991 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 23d5269a5c8c4c049265e18bff4e2c45 (table default.Table_342a368a27a14267af730e53365d41d1 [id=a638366ce0254730b9edc43f81dcb340]) successfully deleted
I20250112 20:51:36.321362 1184 tablet_replica.cc:331] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.321514 1745 leader_election.cc:290] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.321964 1184 raft_consensus.cc:2238] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.322196 1184 raft_consensus.cc:2267] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.322299 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe28f23448c47d3979f1982f939e39c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.322512 1048 raft_consensus.cc:2463] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:36.322569 710 leader_election.cc:343] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:36.322847 1184 ts_tablet_manager.cc:1905] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.322978 710 leader_election.cc:304] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.323454 1745 raft_consensus.cc:2798] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.323580 1745 raft_consensus.cc:491] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.323666 1745 raft_consensus.cc:3054] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.324173 1164 tablet_bootstrap.cc:654] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.326875 1163 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.327033 1163 ts_tablet_manager.cc:1397] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.000s
I20250112 20:51:36.327827 1163 raft_consensus.cc:357] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.328047 1163 raft_consensus.cc:383] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.328153 1163 raft_consensus.cc:738] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.328375 1163 consensus_queue.cc:260] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.329170 1163 ts_tablet_manager.cc:1428] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:36.329686 1163 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.331269 1163 tablet_bootstrap.cc:654] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.332938 1164 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.334302 1164 ts_tablet_manager.cc:1397] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20250112 20:51:36.335685 1164 raft_consensus.cc:357] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.340261 1184 ts_tablet_manager.cc:1918] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.341385 1164 raft_consensus.cc:383] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.341650 1164 raft_consensus.cc:738] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.342052 1184 log.cc:1198] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/af1aac31fd32400b9094f8e2de4fe42c
I20250112 20:51:36.342025 1164 consensus_queue.cc:260] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.342470 1184 ts_tablet_manager.cc:1939] T af1aac31fd32400b9094f8e2de4fe42c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.343092 1164 ts_tablet_manager.cc:1428] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:36.343312 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 448) took 1295 ms. Trace:
I20250112 20:51:36.343731 1164 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.345731 1164 tablet_bootstrap.cc:654] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.346114 1163 tablet_bootstrap.cc:492] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.351186 1163 ts_tablet_manager.cc:1397] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20250112 20:51:36.352248 1164 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.352573 1163 raft_consensus.cc:357] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.352922 1163 raft_consensus.cc:383] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.353020 1163 raft_consensus.cc:738] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.353324 1163 consensus_queue.cc:260] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.353716 1164 ts_tablet_manager.cc:1397] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20250112 20:51:36.353895 1163 ts_tablet_manager.cc:1428] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:36.353989 1184 rpcz_store.cc:276] 0112 20:51:35.047486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:35.050079 (+ 2593us) service_pool.cc:225] Handling call
0112 20:51:36.321305 (+1271226us) ts_tablet_manager.cc:1145] Deleting tablet af1aac31fd32400b9094f8e2de4fe42c
0112 20:51:36.321314 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.331369 (+ 10055us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.340139 (+ 8770us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.343224 (+ 3085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1270962}
I20250112 20:51:36.354641 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet af1aac31fd32400b9094f8e2de4fe42c (table default.Table_be025c70ef21488baea67cc8bd9c2d3d [id=5300ef7da9b942e4a77573e1ab50d41d]) successfully deleted
I20250112 20:51:36.355682 1164 raft_consensus.cc:357] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.357138 1164 raft_consensus.cc:383] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.358237 1164 raft_consensus.cc:738] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.358645 1164 consensus_queue.cc:260] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.360062 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1d97ac93467a4a7a8ae44d52719f191c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d97ac93467a4a7a8ae44d52719f191c already in progress: creating tablet
I20250112 20:51:36.359800 1164 ts_tablet_manager.cc:1428] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:36.361090 1164 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:36.361130 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d7117edf078848c0a1c83ab8e804dfed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7117edf078848c0a1c83ab8e804dfed already in progress: creating tablet
W20250112 20:51:36.361387 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 already in progress: creating tablet
W20250112 20:51:36.361660 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d17e6d263014448b96c5d1f03013e6e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17e6d263014448b96c5d1f03013e6e7 already in progress: creating tablet
W20250112 20:51:36.361917 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1d60604713524dba8f3a42c4f3bc7300 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d60604713524dba8f3a42c4f3bc7300 already in progress: creating tablet
W20250112 20:51:36.362190 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9570269368e14b1abeaffdcc741c6f3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9570269368e14b1abeaffdcc741c6f3a already in progress: creating tablet
I20250112 20:51:36.362388 1163 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.361271 1745 raft_consensus.cc:513] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.363782 710 leader_election.cc:343] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:36.364032 1745 leader_election.cc:290] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.364437 1163 tablet_bootstrap.cc:654] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.364537 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe28f23448c47d3979f1982f939e39c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.364764 1048 raft_consensus.cc:3054] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.365374 1184 tablet_replica.cc:331] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.365592 1184 raft_consensus.cc:2238] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.365762 1184 raft_consensus.cc:2267] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.366276 1184 ts_tablet_manager.cc:1905] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.367569 1048 raft_consensus.cc:2463] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.368031 710 leader_election.cc:304] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.368608 1745 raft_consensus.cc:2798] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.368877 1745 raft_consensus.cc:695] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:36.369203 1745 consensus_queue.cc:237] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.371829 1164 tablet_bootstrap.cc:654] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.376873 1184 ts_tablet_manager.cc:1918] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.380424 1164 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.380620 1164 ts_tablet_manager.cc:1397] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20250112 20:51:36.380940 1184 log.cc:1198] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/201a7e2f17bb46fa918dad8f878a4065
I20250112 20:51:36.381369 1184 ts_tablet_manager.cc:1939] T 201a7e2f17bb46fa918dad8f878a4065 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.381639 1163 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.381803 1163 ts_tablet_manager.cc:1397] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20250112 20:51:36.382512 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 201a7e2f17bb46fa918dad8f878a4065 (table default.Table_d4f060e4fa3648d6943cc97b9b3b7f51 [id=048aebd5f01046ac977b0f8d73481b8d]) successfully deleted
I20250112 20:51:36.382599 1163 raft_consensus.cc:357] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.382843 1163 raft_consensus.cc:383] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.382575 1164 raft_consensus.cc:357] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.382951 1163 raft_consensus.cc:738] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.383140 1164 raft_consensus.cc:383] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.383176 1163 consensus_queue.cc:260] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.383356 1164 raft_consensus.cc:738] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.383852 1163 ts_tablet_manager.cc:1428] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:36.384143 1164 consensus_queue.cc:260] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.384295 1163 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.384804 1184 tablet_replica.cc:331] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.385983 1164 ts_tablet_manager.cc:1428] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:36.386584 1164 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.387686 1184 raft_consensus.cc:2238] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.387962 1184 raft_consensus.cc:2267] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.388676 1184 ts_tablet_manager.cc:1905] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.389786 1164 tablet_bootstrap.cc:654] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.399201 1185 tablet_replica.cc:331] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.215864 884 tablet_service.cc:1467] Processing CreateTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 (DEFAULT_TABLE table=default.Table_0b49521c49204a03b257a7b38a4132c2 [id=08d133fd345241bab23273e839ceda6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.400354 1186 log.cc:1198] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:36.400629 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6d73ab981c4e87aa229a49fb3b6913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.400671 1186 ts_tablet_manager.cc:1939] T 4688dd6bbfc046b7bb140206cae9f104 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.401604 1186 tablet_replica.cc:331] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.401763 1186 raft_consensus.cc:2238] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.401930 1186 raft_consensus.cc:2267] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.403674 1163 tablet_bootstrap.cc:654] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.404585 1185 raft_consensus.cc:2238] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.216248 885 tablet_service.cc:1467] Processing CreateTablet for tablet 61964ec077094c018c5a03ad7699c9c6 (DEFAULT_TABLE table=default.Table_743de41614e248bd9cffca489bc49570 [id=e8f7a569b2914e54b0819c8387667edf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:36.218778 847 leader_election.cc:343] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.404902 847 leader_election.cc:304] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.405069 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61964ec077094c018c5a03ad7699c9c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:36.405229 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 186449us
I20250112 20:51:36.406260 1942 raft_consensus.cc:2743] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.406517 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4688dd6bbfc046b7bb140206cae9f104 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]) successfully deleted
I20250112 20:51:36.406769 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.411310 1184 ts_tablet_manager.cc:1918] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.411448 1184 log.cc:1198] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e72fda1743fc4f96b29add6e36e884de
I20250112 20:51:36.411888 1184 ts_tablet_manager.cc:1939] T e72fda1743fc4f96b29add6e36e884de P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.412859 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e72fda1743fc4f96b29add6e36e884de (table default.Table_15828ef1bbbc41068bb6b6f2fbb18f94 [id=77c2c152845745ad8c8898a9547975d1]) successfully deleted
I20250112 20:51:36.413024 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 65784493c6ef417993b1f6371cab47d6
I20250112 20:51:36.414443 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.415812 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8f636df32bd34438bb54989736eaa431 (table default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]) successfully deleted
I20250112 20:51:36.416394 2052 raft_consensus.cc:491] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.416548 2052 raft_consensus.cc:513] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.416713 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 98547168378947df885c601ccdc85e57
I20250112 20:51:36.417111 2052 leader_election.cc:290] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.417516 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f78d3c2c77dc45bb87a6c4e69bfe161e" } new_table_name: "default.Table_5d0ef950ae7941faa216af8392ab9a72" modify_external_catalogs: true
I20250112 20:51:36.418556 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.419458 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:36.420575 710 leader_election.cc:343] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.421301 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.423589 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c
I20250112 20:51:36.424027 447 catalog_manager.cc:6596] ReplaceTablet: tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c deleted and replaced by tablet dfc63eff44414906a027a8208b5cc5a9
I20250112 20:51:36.424299 1185 raft_consensus.cc:2267] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.412765 1184 tablet_replica.cc:331] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.424465 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_57a800cd62ef4adea24037854c5f479f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.424770 1184 raft_consensus.cc:2238] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:36.424978 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57a800cd62ef4adea24037854c5f479f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.425038 1184 raft_consensus.cc:2267] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.424993 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.420610 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.425348 1048 raft_consensus.cc:2463] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:36.425593 1184 ts_tablet_manager.cc:1905] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.400205 1178 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.425731 710 leader_election.cc:304] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.425853 1178 ts_tablet_manager.cc:1397] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.392s user 0.004s sys 0.007s
I20250112 20:51:36.426182 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.426396 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.426657 2052 raft_consensus.cc:2798] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.426556 1178 raft_consensus.cc:357] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.426753 1178 raft_consensus.cc:383] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.426749 2052 raft_consensus.cc:491] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.426803 1178 raft_consensus.cc:738] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.426854 2052 raft_consensus.cc:3054] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.429785 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.430938 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5dbbb84fc94a4e88bb3331103463e4da" } new_table_name: "default.Table_bf01bbcaf340447098cdbe875519b5e9" modify_external_catalogs: true
I20250112 20:51:36.432922 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.434494 1164 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.434648 1164 ts_tablet_manager.cc:1397] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.004s
I20250112 20:51:36.435348 1164 raft_consensus.cc:357] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.435597 1164 raft_consensus.cc:383] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.435662 1164 raft_consensus.cc:738] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.435850 1164 consensus_queue.cc:260] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.436005 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b37b00c49f234dc89d63f644febc23fb
I20250112 20:51:36.436926 1186 ts_tablet_manager.cc:1905] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.437259 884 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.436277 1164 ts_tablet_manager.cc:1428] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:36.437631 1178 consensus_queue.cc:260] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.438046 1164 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.438153 1178 ts_tablet_manager.cc:1428] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:36.438951 2052 raft_consensus.cc:513] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.439702 1178 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.441181 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:36.440354 2052 leader_election.cc:290] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.441946 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.442119 1048 raft_consensus.cc:3054] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:36.442054 710 leader_election.cc:343] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.436758 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ceaaef5837a2461f90a9ca600452abf6" } modify_external_catalogs: true
I20250112 20:51:36.442629 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ceaaef5837a2461f90a9ca600452abf6" } modify_external_catalogs: true
I20250112 20:51:36.444556 1048 raft_consensus.cc:2463] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.437083 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4ce4f31cf8f24e35b7281d4a8b5667a0
I20250112 20:51:36.444991 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.445081 710 leader_election.cc:304] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.445286 1185 ts_tablet_manager.cc:1905] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.437322 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7f807071e53a4f0e8629b1dfa6d26b9d
I20250112 20:51:36.446420 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c8ca9356c6d94a34b8bca0f720df2028" } new_table_name: "default.Table_92f355d4058f40b6b0979ff3f3f1af14" modify_external_catalogs: true
I20250112 20:51:36.446794 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.447059 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.447441 1178 tablet_bootstrap.cc:654] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.449015 448 catalog_manager.cc:6596] ReplaceTablet: tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 deleted and replaced by tablet 53fcb48fe5314957ae807902274ef596
I20250112 20:51:36.449433 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_72a560b4d6274b2c9fc00315582a7894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.449975 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72a560b4d6274b2c9fc00315582a7894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.450560 1184 ts_tablet_manager.cc:1918] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:36.450678 1184 log.cc:1198] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8aaa5246382745ecbea4b9bd4825e8f1
I20250112 20:51:36.451095 1178 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.451130 1184 ts_tablet_manager.cc:1939] T 8aaa5246382745ecbea4b9bd4825e8f1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.451220 1178 ts_tablet_manager.cc:1397] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250112 20:51:36.451871 1178 raft_consensus.cc:357] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.452051 1178 raft_consensus.cc:383] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.452050 1745 raft_consensus.cc:2798] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.452103 1178 raft_consensus.cc:738] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.452286 1184 tablet_replica.cc:331] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.452288 1745 raft_consensus.cc:695] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:36.452507 1184 raft_consensus.cc:2238] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.452644 1184 raft_consensus.cc:2267] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.452586 1745 consensus_queue.cc:237] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.457844 1178 consensus_queue.cc:260] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.458302 1178 ts_tablet_manager.cc:1428] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
W20250112 20:51:36.461738 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38cdb9a415f745cbb62f27763e490d5b consensus state has no leader
I20250112 20:51:36.462087 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a7e6ee6b896447d3aa166e1d4cfb2d79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.462584 1186 ts_tablet_manager.cc:1918] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250112 20:51:36.462612 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7e6ee6b896447d3aa166e1d4cfb2d79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.462666 1186 log.cc:1198] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9c55b922c39545ac9c89115cd587ea8c
I20250112 20:51:36.464344 1185 ts_tablet_manager.cc:1918] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.464484 1185 log.cc:1198] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/64cd83b199bb4da28d39da449945c389
I20250112 20:51:36.464933 1185 ts_tablet_manager.cc:1939] T 64cd83b199bb4da28d39da449945c389 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.464932 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 691e2c3f72ab4f439678d166b9b31661 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.465381 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8aaa5246382745ecbea4b9bd4825e8f1 (table default.Table_15e3fd8e71984a81acb2dabf8f3563f7 [id=9a97eeefbfac4dc2bb4a8ceab84304b3]) successfully deleted
W20250112 20:51:36.465636 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4c3f359655c94f1d905943069548845b because the tablet was not found. No further retry: Not found: Tablet not found: 4c3f359655c94f1d905943069548845b
I20250112 20:51:36.465824 1185 tablet_replica.cc:331] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.466040 1185 raft_consensus.cc:2238] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.466190 1185 raft_consensus.cc:2267] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.466591 1178 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.466678 1185 ts_tablet_manager.cc:1905] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.466814 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.467098 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.469179 1184 ts_tablet_manager.cc:1905] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.472879 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.477108 1186 ts_tablet_manager.cc:1939] T 9c55b922c39545ac9c89115cd587ea8c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.477164 1178 tablet_bootstrap.cc:654] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.478338 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]) successfully deleted
I20250112 20:51:36.478674 1186 tablet_replica.cc:331] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.478880 1186 raft_consensus.cc:2238] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.479080 1186 raft_consensus.cc:2267] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.479596 1186 ts_tablet_manager.cc:1905] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.481691 1178 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.481828 1178 ts_tablet_manager.cc:1397] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20250112 20:51:36.482551 1178 raft_consensus.cc:357] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.485543 1178 raft_consensus.cc:383] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.485635 1178 raft_consensus.cc:738] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.485837 1821 raft_consensus.cc:491] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.486014 1821 raft_consensus.cc:513] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.486157 1178 consensus_queue.cc:260] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.486696 1178 ts_tablet_manager.cc:1428] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:36.486699 1821 leader_election.cc:290] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.487664 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.488905 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d32367b39a4cce8371377ec7449586" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.489537 983 leader_election.cc:304] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.489998 1821 raft_consensus.cc:2743] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:36.491593 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db4fa71cf46144629e1fc1f2172fc3cb consensus state has no leader
I20250112 20:51:36.493826 447 master_service.cc:946] ReplaceTablet: received request to replace tablet ad1a4b5ac2f94874b99b3ef790a034fd from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.494086 1178 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.494551 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5cf59aa9a65542c1b2bab56c2238c6a7" } new_table_name: "default.Table_e2d706e7614440a1905f8790f025ac29" modify_external_catalogs: true
I20250112 20:51:36.496366 1178 tablet_bootstrap.cc:654] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.502470 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:36.503551 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_dcdefc016f7546e39891ebe15594cba1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.504318 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcdefc016f7546e39891ebe15594cba1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:36.505884 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8bea4d83ea644a0930a29f61e7a5fc8 consensus state has no leader
I20250112 20:51:36.506392 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3e98d1a51a4e4971afa54b9847d7888b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.506934 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e98d1a51a4e4971afa54b9847d7888b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.509801 1184 ts_tablet_manager.cc:1918] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.510967 1185 ts_tablet_manager.cc:1918] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.511220 1184 log.cc:1198] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/83c13126e53b46c0a1e7beafe1b0d295
I20250112 20:51:36.514905 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 64cd83b199bb4da28d39da449945c389 (table default.Table_0afc27e58eec42daadf528a66340c624 [id=370444777ab4411f837c19bc3824376a]) successfully deleted
I20250112 20:51:36.515163 1184 ts_tablet_manager.cc:1939] T 83c13126e53b46c0a1e7beafe1b0d295 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.516372 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 83c13126e53b46c0a1e7beafe1b0d295 (table default.Table_c62b92e5784e44af9e27e902aa48a6ca [id=212d41ddb2d248a0bcf3aa02f7b10bcd]) successfully deleted
I20250112 20:51:36.517338 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0366d760e34a78b9144e6d16a3db40 (DEFAULT_TABLE table=default.Table_6b593fd0b27949e2b15bfc7da5a989be [id=0311dc32bc044029bed011774edac805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.517729 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0366d760e34a78b9144e6d16a3db40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.512854 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.513237 739 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.518131 1185 log.cc:1198] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/83c13126e53b46c0a1e7beafe1b0d295
I20250112 20:51:36.518968 1186 ts_tablet_manager.cc:1918] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.519215 1185 ts_tablet_manager.cc:1939] T 83c13126e53b46c0a1e7beafe1b0d295 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.521075 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 (DEFAULT_TABLE table=default.Table_e9dd5f9cbc524e449863f6c9b52424b6 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.521510 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f6ad2ac50c745448031bdf94e0eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.521894 1185 tablet_replica.cc:331] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.520502 1186 log.cc:1198] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7deb6e37f75e4aebacb04c4553049940
I20250112 20:51:36.522809 1186 ts_tablet_manager.cc:1939] T 7deb6e37f75e4aebacb04c4553049940 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.523280 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]). Expected version 1 got 0
I20250112 20:51:36.523707 1184 tablet_replica.cc:331] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.523883 1184 raft_consensus.cc:2238] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.524614 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e44422f75a744d078cdf4f0997242e22
I20250112 20:51:36.513517 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.514103 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.525507 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_09c966930c7b4277a84b783534270688" } new_table_name: "default.Table_de5c34db50e64848a3fb7a3d1a4e3613" modify_external_catalogs: true
I20250112 20:51:36.526098 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e44422f75a744d078cdf4f0997242e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.526314 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet e44422f75a744d078cdf4f0997242e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.525614 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7deb6e37f75e4aebacb04c4553049940 (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
W20250112 20:51:36.527139 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9880e7d303b54d959bca4e8464a6b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9880e7d303b54d959bca4e8464a6b30c already in progress: creating tablet
W20250112 20:51:36.527403 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efa7f6a1e9804e6ab822f9d4c004b1b3 already in progress: creating tablet
I20250112 20:51:36.528892 1184 pending_rounds.cc:62] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:36.529032 1184 pending_rounds.cc:69] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585032031281152 op_type: NO_OP noop_request { }
I20250112 20:51:36.529225 1184 raft_consensus.cc:2883] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:36.529362 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2f7947f8e7af4b3799d132d1f226f57a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.530054 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f7947f8e7af4b3799d132d1f226f57a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.531900 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2a1cc8e9bac14fc7ab9057e1dd447fb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.532405 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a1cc8e9bac14fc7ab9057e1dd447fb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.531785 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4688dd6bbfc046b7bb140206cae9f104 with delete_type TABLET_DATA_DELETED (replaced by tablet d477fb5e5ad5447bb182bc9998561a43) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.531956 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.533071 1184 raft_consensus.cc:2267] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.533428 1186 tablet_replica.cc:331] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.534225 1184 ts_tablet_manager.cc:1905] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.534646 1185 raft_consensus.cc:2238] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.534822 1186 raft_consensus.cc:2238] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.535049 1185 raft_consensus.cc:2267] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.535254 1186 raft_consensus.cc:2267] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.535383 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_669829b96b0a4f1fab410ba1e9b1e5d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.535825 1185 ts_tablet_manager.cc:1905] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:36.536124 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_669829b96b0a4f1fab410ba1e9b1e5d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.537807 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 (DEFAULT_TABLE table=default.Table_e9dd5f9cbc524e449863f6c9b52424b6 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.539893 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1968191ac14099b6035ac487e3e6bb (DEFAULT_TABLE table=default.Table_037b9792e05d4bd08c1ae042e95cba5f [id=74fa34ad178445e4959baab29aa2f90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.540409 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1968191ac14099b6035ac487e3e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.540628 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f6ad2ac50c745448031bdf94e0eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.544412 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b7f79b3309e45afb9e18cad97ddfeb0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.555019 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 83c13126e53b46c0a1e7beafe1b0d295 (table default.Table_c62b92e5784e44af9e27e902aa48a6ca [id=212d41ddb2d248a0bcf3aa02f7b10bcd]) successfully deleted
W20250112 20:51:36.556469 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d8330fa1e4654a1a9f167a1120a103fc because the tablet was not found. No further retry: Not found: Tablet not found: d8330fa1e4654a1a9f167a1120a103fc
W20250112 20:51:36.556948 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9880e7d303b54d959bca4e8464a6b30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9880e7d303b54d959bca4e8464a6b30c already in progress: creating tablet
W20250112 20:51:36.557724 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efa7f6a1e9804e6ab822f9d4c004b1b3 already in progress: creating tablet
W20250112 20:51:36.558151 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5d167065029a46b791d413db8a2c4bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d167065029a46b791d413db8a2c4bcc already in progress: creating tablet
W20250112 20:51:36.558898 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 921ae136d5e2438e950b96f156f23da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 921ae136d5e2438e950b96f156f23da3 already in progress: creating tablet
I20250112 20:51:36.560271 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 (DEFAULT_TABLE table=default.Table_cea4e75bb1d34fcdb209c15b1cd6738d [id=673551cb83594e2a9b83aa59fd498017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.537879 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0366d760e34a78b9144e6d16a3db40 (DEFAULT_TABLE table=default.Table_6b593fd0b27949e2b15bfc7da5a989be [id=0311dc32bc044029bed011774edac805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.560612 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a11203ca15745c0a89a448e1cfc2cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.560837 1186 ts_tablet_manager.cc:1905] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.538688 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1968191ac14099b6035ac487e3e6bb (DEFAULT_TABLE table=default.Table_037b9792e05d4bd08c1ae042e95cba5f [id=74fa34ad178445e4959baab29aa2f90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.561152 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0366d760e34a78b9144e6d16a3db40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.565054 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1968191ac14099b6035ac487e3e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:36.565356 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9854d410371b442c81705698f427d8b0 consensus state has no leader
I20250112 20:51:36.570784 1186 ts_tablet_manager.cc:1918] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.570885 1186 log.cc:1198] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:36.539119 880 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.542855 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.543252 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.543648 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.546249 877 tablet_service.cc:1467] Processing CreateTablet for tablet e3648997eb1241edaf56c825bf76b240 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.571434 1745 raft_consensus.cc:491] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.571492 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648997eb1241edaf56c825bf76b240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.548233 888 tablet_service.cc:1467] Processing CreateTablet for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9 (DEFAULT_TABLE table=default.Table_ca3bce77c81b44b98c37ce00f1536286 [id=0120f976e8ee427daf8d55cb42ce7463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.575922 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.552403 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 (DEFAULT_TABLE table=default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a [id=56be4f07e7724011bbe3b2989d6ff5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.441247 1164 tablet_bootstrap.cc:654] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.580436 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8c7784e5534c2c91f124dffd24d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.582309 1184 ts_tablet_manager.cc:1918] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.582427 1184 log.cc:1198] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7deb6e37f75e4aebacb04c4553049940
I20250112 20:51:36.582773 1184 ts_tablet_manager.cc:1939] T 7deb6e37f75e4aebacb04c4553049940 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.583673 1184 tablet_replica.cc:331] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.583864 1184 raft_consensus.cc:2238] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.584057 1184 pending_rounds.cc:62] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:36.584093 1184 pending_rounds.cc:69] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585031707123712 op_type: NO_OP noop_request { }
I20250112 20:51:36.584177 1184 raft_consensus.cc:2883] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:36.584241 1184 raft_consensus.cc:2267] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.554152 878 tablet_service.cc:1467] Processing CreateTablet for tablet be6751397eab403a8998a6a71820a6f9 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.584903 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be6751397eab403a8998a6a71820a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.585006 1178 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.585108 1178 ts_tablet_manager.cc:1397] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.018s
I20250112 20:51:36.585776 1178 raft_consensus.cc:357] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.585937 1178 raft_consensus.cc:383] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.585979 1178 raft_consensus.cc:738] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.586153 1178 consensus_queue.cc:260] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.586589 1178 ts_tablet_manager.cc:1428] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:36.586786 1186 ts_tablet_manager.cc:1939] T 42eb0f79df5444f8a8360921d5f2b9ee P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.588060 1178 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.556046 890 tablet_service.cc:1467] Processing CreateTablet for tablet a5819f1c6329433183492855c3b08476 (DEFAULT_TABLE table=default.Table_557ef80f7a9c4980aa3aa81e9000ddf7 [id=f10d36396bd54b209388fc097227ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.588598 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5819f1c6329433183492855c3b08476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.593235 2076 raft_consensus.cc:491] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.593415 2076 raft_consensus.cc:513] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.593832 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0366d760e34a78b9144e6d16a3db40 (DEFAULT_TABLE table=default.Table_6b593fd0b27949e2b15bfc7da5a989be [id=0311dc32bc044029bed011774edac805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.594041 2076 leader_election.cc:290] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.571549 1745 raft_consensus.cc:513] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.594790 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.595244 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0366d760e34a78b9144e6d16a3db40. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:36.595465 847 leader_election.cc:343] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.594645 1186 tablet_replica.cc:331] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.596372 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.596626 1186 raft_consensus.cc:2238] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.596930 1186 raft_consensus.cc:2267] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.597890 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01367a11dea04b33a9255508501b9060" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.600118 1745 leader_election.cc:290] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.600119 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01367a11dea04b33a9255508501b9060" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.600304 1048 raft_consensus.cc:2463] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:36.600580 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 42eb0f79df5444f8a8360921d5f2b9ee (table default.Table_43f4ea15650c4619afdabdb3c169df86 [id=dd64a2679c0849279e9dae49f268056b]) successfully deleted
I20250112 20:51:36.596550 775 raft_consensus.cc:2463] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:36.600926 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 65784493c6ef417993b1f6371cab47d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65784493c6ef417993b1f6371cab47d6 already in progress: creating tablet
W20250112 20:51:36.601137 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 98547168378947df885c601ccdc85e57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98547168378947df885c601ccdc85e57 already in progress: creating tablet
W20250112 20:51:36.601325 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c already in progress: creating tablet
W20250112 20:51:36.601486 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b37b00c49f234dc89d63f644febc23fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37b00c49f234dc89d63f644febc23fb already in progress: creating tablet
W20250112 20:51:36.601675 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 7f807071e53a4f0e8629b1dfa6d26b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f807071e53a4f0e8629b1dfa6d26b9d already in progress: creating tablet
W20250112 20:51:36.601871 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 already in progress: creating tablet
W20250112 20:51:36.602152 710 leader_election.cc:343] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.603757 1185 ts_tablet_manager.cc:1918] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.603870 1185 log.cc:1198] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7deb6e37f75e4aebacb04c4553049940
I20250112 20:51:36.604161 1185 ts_tablet_manager.cc:1939] T 7deb6e37f75e4aebacb04c4553049940 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.605039 1010 tablet_service.cc:1467] Processing CreateTablet for tablet e3648997eb1241edaf56c825bf76b240 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.605391 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648997eb1241edaf56c825bf76b240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.605489 1184 ts_tablet_manager.cc:1905] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.606685 847 leader_election.cc:304] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:36.607059 2076 raft_consensus.cc:491] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.607154 2076 raft_consensus.cc:513] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.607704 2076 leader_election.cc:290] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.607883 2076 raft_consensus.cc:2798] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.607947 2076 raft_consensus.cc:491] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.608013 2076 raft_consensus.cc:3054] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.608457 1185 tablet_replica.cc:331] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.608628 1185 raft_consensus.cc:2238] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.608767 1185 raft_consensus.cc:2267] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.609262 1185 ts_tablet_manager.cc:1905] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.611122 2076 raft_consensus.cc:513] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.611670 2076 leader_election.cc:290] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.612272 1178 tablet_bootstrap.cc:654] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.599622 1186 ts_tablet_manager.cc:1905] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.613046 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.613137 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9 (DEFAULT_TABLE table=default.Table_ca3bce77c81b44b98c37ce00f1536286 [id=0120f976e8ee427daf8d55cb42ce7463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.613178 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54de54eabe5946058d1940b79f5fb832" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.613348 775 raft_consensus.cc:2463] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:36.613185 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71743c73b6f24da9b14a3d4148117813" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.613678 770 raft_consensus.cc:3054] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.613639 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54de54eabe5946058d1940b79f5fb832" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.614269 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 (DEFAULT_TABLE table=default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a [id=56be4f07e7724011bbe3b2989d6ff5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.615671 1025 tablet_service.cc:1467] Processing CreateTablet for tablet be6751397eab403a8998a6a71820a6f9 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.616025 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 691e2c3f72ab4f439678d166b9b31661
I20250112 20:51:36.616113 770 raft_consensus.cc:2463] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:36.616276 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e44422f75a744d078cdf4f0997242e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.616523 446 catalog_manager.cc:6596] ReplaceTablet: tablet 691e2c3f72ab4f439678d166b9b31661 deleted and replaced by tablet 69d57977510c4422ad625a7045ede7e4
I20250112 20:51:36.617096 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a5819f1c6329433183492855c3b08476 (DEFAULT_TABLE table=default.Table_557ef80f7a9c4980aa3aa81e9000ddf7 [id=f10d36396bd54b209388fc097227ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.617288 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.617336 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c456c3048d45444f9422bf7a190b290a" } modify_external_catalogs: true
I20250112 20:51:36.617527 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c456c3048d45444f9422bf7a190b290a" } modify_external_catalogs: true
I20250112 20:51:36.618569 710 leader_election.cc:304] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.618921 1745 raft_consensus.cc:2798] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.618872 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 (DEFAULT_TABLE table=default.Table_e9dd5f9cbc524e449863f6c9b52424b6 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.619004 1745 raft_consensus.cc:491] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.619067 1745 raft_consensus.cc:3054] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.619199 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f6ad2ac50c745448031bdf94e0eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.617293 847 leader_election.cc:304] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.620839 847 leader_election.cc:304] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:36.621068 847 leader_election.cc:343] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:36.621318 847 leader_election.cc:343] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.621557 1942 raft_consensus.cc:2798] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.621620 1942 raft_consensus.cc:491] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.621675 1942 raft_consensus.cc:3054] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.621569 1745 raft_consensus.cc:513] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.622831 1982 raft_consensus.cc:2798] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.623260 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01367a11dea04b33a9255508501b9060" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:36.623744 710 leader_election.cc:343] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.624073 1942 raft_consensus.cc:513] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.624639 1942 leader_election.cc:290] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.625154 1982 raft_consensus.cc:695] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:36.625311 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 104d6b4effb84314a6b2b9db883b4366
I20250112 20:51:36.625404 1982 consensus_queue.cc:237] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.625821 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1968191ac14099b6035ac487e3e6bb (DEFAULT_TABLE table=default.Table_037b9792e05d4bd08c1ae042e95cba5f [id=74fa34ad178445e4959baab29aa2f90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.626379 1745 leader_election.cc:290] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.626487 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54de54eabe5946058d1940b79f5fb832" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.627156 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54de54eabe5946058d1940b79f5fb832" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.627398 770 raft_consensus.cc:3054] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.628314 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e1968191ac14099b6035ac487e3e6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.633023 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01367a11dea04b33a9255508501b9060" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:36.633050 847 leader_election.cc:343] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.633208 1048 raft_consensus.cc:3054] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.636050 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 64cd83b199bb4da28d39da449945c389 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.636916 1048 raft_consensus.cc:2463] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.638012 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.638624 741 tablet_service.cc:1467] Processing CreateTablet for tablet e3648997eb1241edaf56c825bf76b240 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.639003 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648997eb1241edaf56c825bf76b240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.639024 710 leader_election.cc:304] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.639456 1745 raft_consensus.cc:2798] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.639706 1745 raft_consensus.cc:695] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:36.639964 1745 consensus_queue.cc:237] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.645249 450 catalog_manager.cc:5526] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:36.645747 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]). Expected version 1 got 0
I20250112 20:51:36.648691 741 tablet_service.cc:1467] Processing CreateTablet for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9 (DEFAULT_TABLE table=default.Table_ca3bce77c81b44b98c37ce00f1536286 [id=0120f976e8ee427daf8d55cb42ce7463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.649092 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.649127 1178 tablet_bootstrap.cc:492] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.649428 1178 ts_tablet_manager.cc:1397] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20250112 20:51:36.650926 1178 raft_consensus.cc:357] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.652933 770 raft_consensus.cc:2463] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:36.653268 1178 raft_consensus.cc:383] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.653342 1178 raft_consensus.cc:738] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.653514 1178 consensus_queue.cc:260] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.653995 1178 ts_tablet_manager.cc:1428] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:36.656337 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.659322 1178 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.661842 884 tablet_service.cc:1467] Processing CreateTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a (DEFAULT_TABLE table=default.Table_df6048c1d029498b8d19b7645fa5afd8 [id=a6d15af404b545bcb85000a8d0bf38f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.662020 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 (DEFAULT_TABLE table=default.Table_cea4e75bb1d34fcdb209c15b1cd6738d [id=673551cb83594e2a9b83aa59fd498017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.662503 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.655308 739 tablet_service.cc:1467] Processing CreateTablet for tablet be6751397eab403a8998a6a71820a6f9 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.655356 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655545 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655715 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655866 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655922 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655995 749 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.656266 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 (DEFAULT_TABLE table=default.Table_cea4e75bb1d34fcdb209c15b1cd6738d [id=673551cb83594e2a9b83aa59fd498017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.656453 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.655524 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.656663 754 tablet_service.cc:1467] Processing CreateTablet for tablet a5819f1c6329433183492855c3b08476 (DEFAULT_TABLE table=default.Table_557ef80f7a9c4980aa3aa81e9000ddf7 [id=f10d36396bd54b209388fc097227ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.665997 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0fd03aba45a64415a098af438c178bc5" } modify_external_catalogs: true
I20250112 20:51:36.666205 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0fd03aba45a64415a098af438c178bc5" } modify_external_catalogs: true
I20250112 20:51:36.666421 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5819f1c6329433183492855c3b08476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.666594 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.666908 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a11203ca15745c0a89a448e1cfc2cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.667537 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_47bcd1dceb5441d4a1a05819fb432d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.668099 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47bcd1dceb5441d4a1a05819fb432d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.668740 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe28f23448c47d3979f1982f939e39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:27 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.669059 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be6751397eab403a8998a6a71820a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.669579 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_141fc45760534923b67d082dae64762d" } modify_external_catalogs: true
I20250112 20:51:36.669731 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_141fc45760534923b67d082dae64762d" } modify_external_catalogs: true
I20250112 20:51:36.670797 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9" } modify_external_catalogs: true
I20250112 20:51:36.670929 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9" } modify_external_catalogs: true
I20250112 20:51:36.672967 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8c7784e5534c2c91f124dffd24d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.675851 1047 raft_consensus.cc:3054] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.677042 847 leader_election.cc:304] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:36.677402 2076 raft_consensus.cc:2798] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.677510 2076 raft_consensus.cc:695] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:36.677713 2076 consensus_queue.cc:237] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.682411 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7deb6e37f75e4aebacb04c4553049940 (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
I20250112 20:51:36.683790 1178 tablet_bootstrap.cc:654] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.683671 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3fc149714fca4476ae1e1091f86a1fcc" } new_table_name: "default.Table_c0ad7aebc1e245b2bcd5c4af8c8211c6" modify_external_catalogs: true
I20250112 20:51:36.683832 1025 tablet_service.cc:1467] Processing CreateTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a (DEFAULT_TABLE table=default.Table_df6048c1d029498b8d19b7645fa5afd8 [id=a6d15af404b545bcb85000a8d0bf38f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.684140 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1dd81dcdc94420bb12a34c1167da8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.685171 1186 ts_tablet_manager.cc:1918] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.685937 1186 log.cc:1198] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e44422f75a744d078cdf4f0997242e22
I20250112 20:51:36.686424 1186 ts_tablet_manager.cc:1939] T e44422f75a744d078cdf4f0997242e22 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.688216 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7deb6e37f75e4aebacb04c4553049940 (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
I20250112 20:51:36.687124 1178 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.687610 880 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ad7cf1e0c74265a4d21931279f2318 with delete_type TABLET_DATA_DELETED (replaced by tablet 35dcd1a120684446bbe19303fb8416ef) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.689272 1178 ts_tablet_manager.cc:1397] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20250112 20:51:36.691243 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 577dc789f0cb48f49168124b37a6fba0
I20250112 20:51:36.691493 1178 raft_consensus.cc:357] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.691654 1178 raft_consensus.cc:383] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.691695 1178 raft_consensus.cc:738] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.691876 1178 consensus_queue.cc:260] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.692245 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b532068d2374b59a3c46525db7c50db
I20250112 20:51:36.692313 1178 ts_tablet_manager.cc:1428] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250112 20:51:36.693531 1184 ts_tablet_manager.cc:1918] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.696668 753 tablet_service.cc:1467] Processing CreateTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a (DEFAULT_TABLE table=default.Table_df6048c1d029498b8d19b7645fa5afd8 [id=a6d15af404b545bcb85000a8d0bf38f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.697134 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.697547 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.659072 1185 ts_tablet_manager.cc:1918] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.698292 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.698343 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.660367 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 (DEFAULT_TABLE table=default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a [id=56be4f07e7724011bbe3b2989d6ff5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.663936 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be6751397eab403a8998a6a71820a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.698711 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a11203ca15745c0a89a448e1cfc2cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.700052 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1dd81dcdc94420bb12a34c1167da8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.704160 1982 raft_consensus.cc:491] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.704279 1982 raft_consensus.cc:513] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.704887 1982 leader_election.cc:290] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.705247 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cae067df1a40098f81cfbd17224c75" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:36.705799 847 leader_election.cc:343] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.705925 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5819f1c6329433183492855c3b08476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.709043 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8c7784e5534c2c91f124dffd24d5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.713424 1185 log.cc:1198] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:36.713748 1185 ts_tablet_manager.cc:1939] T 4688dd6bbfc046b7bb140206cae9f104 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.714614 1185 tablet_replica.cc:331] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.714854 1185 raft_consensus.cc:2238] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.715215 1185 raft_consensus.cc:2267] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.715921 1185 ts_tablet_manager.cc:1905] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.717157 1982 raft_consensus.cc:491] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.717290 1982 raft_consensus.cc:513] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.717765 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4688dd6bbfc046b7bb140206cae9f104 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]) successfully deleted
I20250112 20:51:36.717859 1982 leader_election.cc:290] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:36.721019 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ef01b41b6140d6bdad07bc79dd7dc5 already in progress: creating tablet
I20250112 20:51:36.721553 1745 consensus_queue.cc:1035] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:36.696576 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.705451 1821 raft_consensus.cc:491] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.727066 1821 raft_consensus.cc:513] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:36.727805 1821 leader_election.cc:290] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.705653 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cae067df1a40098f81cfbd17224c75" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.728593 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:36.728658 847 leader_election.cc:343] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.728842 847 leader_election.cc:304] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.729322 1982 raft_consensus.cc:2743] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.729578 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44422f75a744d078cdf4f0997242e22" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.730055 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8250621cbe45eba669b5ef14e672c2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:36.730082 983 leader_election.cc:343] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250112 20:51:36.730180 983 leader_election.cc:304] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.730281 770 raft_consensus.cc:2463] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:36.730561 1821 raft_consensus.cc:2743] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.730629 847 leader_election.cc:304] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.717243 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.718429 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8250621cbe45eba669b5ef14e672c2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.730885 1982 raft_consensus.cc:2798] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.730945 1982 raft_consensus.cc:491] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.730944 1184 log.cc:1198] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b84624dc685b401192ca753ff59da016
I20250112 20:51:36.731005 1982 raft_consensus.cc:3054] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.719980 1163 tablet_bootstrap.cc:492] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.731289 1184 ts_tablet_manager.cc:1939] T b84624dc685b401192ca753ff59da016 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.731393 1163 ts_tablet_manager.cc:1397] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.347s user 0.023s sys 0.030s
I20250112 20:51:36.732023 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 487) took 1022 ms. Trace:
I20250112 20:51:36.732105 1184 rpcz_store.cc:276] 0112 20:51:35.709594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:35.709648 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:36.583626 (+873978us) ts_tablet_manager.cc:1145] Deleting tablet b84624dc685b401192ca753ff59da016
0112 20:51:36.583636 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.690893 (+107257us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.693467 (+ 2574us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.732006 (+ 38539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":872333}
I20250112 20:51:36.732558 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b84624dc685b401192ca753ff59da016 (table default.Table_ab43d30cd9214becbb707a604ec4b376 [id=afb1164a3af640c7a8626216069a0edb]) successfully deleted
I20250112 20:51:36.732535 1163 raft_consensus.cc:357] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.732693 1163 raft_consensus.cc:383] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.732947 1163 raft_consensus.cc:738] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.733182 1163 consensus_queue.cc:260] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.733484 1982 raft_consensus.cc:513] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.734027 1982 leader_election.cc:290] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.734074 1163 ts_tablet_manager.cc:1428] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250112 20:51:36.734422 847 leader_election.cc:343] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.734516 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8250621cbe45eba669b5ef14e672c2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:36.735934 847 leader_election.cc:343] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.737351 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8250621cbe45eba669b5ef14e672c2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.737506 770 raft_consensus.cc:3054] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.737775 1184 tablet_replica.cc:331] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.738027 1184 raft_consensus.cc:2238] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.738178 1184 raft_consensus.cc:2267] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.741408 1184 ts_tablet_manager.cc:1905] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.744248 770 raft_consensus.cc:2463] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:36.746192 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_42c75ef9c5564cfb835f9aa1a3ad4ecb" } new_table_name: "default.Table_c09ffcd01d594a1d8b065295cf69258a" modify_external_catalogs: true
I20250112 20:51:36.746323 847 leader_election.cc:304] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:36.746744 1982 raft_consensus.cc:2798] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.746906 1982 raft_consensus.cc:695] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:36.747112 1982 consensus_queue.cc:237] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.747627 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43f6a278ed0b4388aceea8a274566af9" } modify_external_catalogs: true
I20250112 20:51:36.747828 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43f6a278ed0b4388aceea8a274566af9" } modify_external_catalogs: true
I20250112 20:51:36.748488 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6b2acbe71aac4af0978c56cb845b5dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.749038 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b2acbe71aac4af0978c56cb845b5dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.750350 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_230aea5bb3074c90aa9e772ab27c1bce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.750955 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_230aea5bb3074c90aa9e772ab27c1bce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.756574 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1dd81dcdc94420bb12a34c1167da8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.756484 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8d0fe81bfac94ece91fa136594e0ae88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.756701 1186 tablet_replica.cc:331] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:36.757136 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5d5545a5c04446e8a4b0cba0ea3ba62a" } new_table_name: "default.Table_ee975b071729460b863016012a403cb4" modify_external_catalogs: true
I20250112 20:51:36.757375 1178 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.758561 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_dae7a8c6178b4a5aab35a17c2abd509f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.759104 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dae7a8c6178b4a5aab35a17c2abd509f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.759409 1178 tablet_bootstrap.cc:654] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.759790 1185 ts_tablet_manager.cc:1918] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:36.759878 1185 log.cc:1198] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9c55b922c39545ac9c89115cd587ea8c
I20250112 20:51:36.760428 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e44422f75a744d078cdf4f0997242e22 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
W20250112 20:51:36.760622 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0fe81bfac94ece91fa136594e0ae88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.766170 1163 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.768832 1164 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.777433 1164 ts_tablet_manager.cc:1397] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.340s user 0.001s sys 0.005s
I20250112 20:51:36.778438 1164 raft_consensus.cc:357] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.778786 1164 raft_consensus.cc:383] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.778900 1164 raft_consensus.cc:738] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.779305 1164 consensus_queue.cc:260] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.770612 770 raft_consensus.cc:1270] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:36.780207 1164 ts_tablet_manager.cc:1428] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:36.780504 1942 consensus_queue.cc:1035] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:36.777035 2131 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:36.782356 847 consensus_peers.cc:487] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:36.784070 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db9bab2477a94c169817d9de8aaad1c0 consensus state has no leader
I20250112 20:51:36.785037 2076 raft_consensus.cc:491] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.785174 2076 raft_consensus.cc:513] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.785833 1186 raft_consensus.cc:2238] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.786278 1186 pending_rounds.cc:62] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:36.786324 1186 pending_rounds.cc:69] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585036279263232 op_type: NO_OP noop_request { }
I20250112 20:51:36.786453 1186 raft_consensus.cc:2883] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:36.786526 1186 raft_consensus.cc:2267] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.786787 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet eda44c48bf9746b59ccdc4d39a41ea91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.787166 1164 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.787312 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572172a274494e429b7e982815897e61" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.787464 770 raft_consensus.cc:2463] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:36.787778 847 leader_election.cc:304] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.788010 2076 leader_election.cc:290] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.788121 2076 raft_consensus.cc:2798] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.788179 2076 raft_consensus.cc:491] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.788247 2076 raft_consensus.cc:3054] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.789043 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:36.789110 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b7b788acf8047acb34ca2596bbf141b consensus state has no leader
I20250112 20:51:36.789598 1164 tablet_bootstrap.cc:654] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.789629 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_db6e2537abb1488faa40d8119ac5f537" } new_table_name: "default.Table_604c0839efdd41cc8400741910f59256" modify_external_catalogs: true
I20250112 20:51:36.790421 1821 raft_consensus.cc:491] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.790535 1821 raft_consensus.cc:513] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.791069 1821 leader_election.cc:290] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.791350 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2734d2857154455f9cbdcd8e5f06b1f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.791606 2076 raft_consensus.cc:513] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.792048 983 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:36.792440 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572172a274494e429b7e982815897e61" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:36.792582 983 leader_election.cc:343] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:36.792661 983 leader_election.cc:304] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.792886 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d706f88f14ab4fd9a459944863afaacd
I20250112 20:51:36.793054 1821 raft_consensus.cc:2743] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:36.793063 847 leader_election.cc:343] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.760185 1185 ts_tablet_manager.cc:1939] T 9c55b922c39545ac9c89115cd587ea8c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.793536 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572172a274494e429b7e982815897e61" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:36.793962 847 leader_election.cc:343] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.794389 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]) successfully deleted
I20250112 20:51:36.795631 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.795750 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.795681 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572172a274494e429b7e982815897e61" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.795889 770 raft_consensus.cc:3054] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.797148 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.798372 770 raft_consensus.cc:2463] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:36.798740 847 leader_election.cc:304] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:36.798864 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 267ebb3c30194d9fa6a092d9bec4eadf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267ebb3c30194d9fa6a092d9bec4eadf already in progress: creating tablet
I20250112 20:51:36.798996 2076 leader_election.cc:290] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.799146 2108 raft_consensus.cc:2798] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.799253 2108 raft_consensus.cc:695] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:36.799489 2108 consensus_queue.cc:237] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.801168 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2734d2857154455f9cbdcd8e5f06b1f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.801208 1185 tablet_replica.cc:331] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.801374 1185 raft_consensus.cc:2238] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.801482 1185 raft_consensus.cc:2267] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.801945 1185 ts_tablet_manager.cc:1905] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.802207 2059 consensus_queue.cc:1035] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:36.803388 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d45ae394dc7849ca9b6f6a8bb647dce0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.803930 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45ae394dc7849ca9b6f6a8bb647dce0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.805914 1184 ts_tablet_manager.cc:1918] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.806084 1184 log.cc:1198] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ac4e304a1212412f8e99bfd222a110b5
I20250112 20:51:36.806427 1184 ts_tablet_manager.cc:1939] T ac4e304a1212412f8e99bfd222a110b5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.807305 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 489) took 1097 ms. Trace:
I20250112 20:51:36.807442 1184 rpcz_store.cc:276] 0112 20:51:35.709870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:35.712703 (+ 2833us) service_pool.cc:225] Handling call
0112 20:51:36.737701 (+1024998us) ts_tablet_manager.cc:1145] Deleting tablet ac4e304a1212412f8e99bfd222a110b5
0112 20:51:36.737714 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.752224 (+ 14510us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.805808 (+ 53584us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:36.807284 (+ 1476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019500}
I20250112 20:51:36.808084 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ac4e304a1212412f8e99bfd222a110b5 (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:36.808737 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7265385e3f78422c91c8622a9305378b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.810267 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7265385e3f78422c91c8622a9305378b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.826283 1164 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.826478 1164 ts_tablet_manager.cc:1397] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250112 20:51:36.828195 1164 raft_consensus.cc:357] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.828548 1164 raft_consensus.cc:383] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.828649 1164 raft_consensus.cc:738] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.828974 1164 consensus_queue.cc:260] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.829888 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11a4b78b73c54798b358bc7647928f4e" } new_table_name: "default.Table_9ff8fdbb27a140239a8286434feff5a2" modify_external_catalogs: true
I20250112 20:51:36.830825 1164 ts_tablet_manager.cc:1428] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:36.831718 1164 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.831944 1186 ts_tablet_manager.cc:1905] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.833168 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 490) took 1120 ms. Trace:
I20250112 20:51:36.833269 1184 rpcz_store.cc:276] 0112 20:51:35.713096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:35.716466 (+ 3370us) service_pool.cc:225] Handling call
0112 20:51:36.833002 (+1116536us) ts_tablet_manager.cc:1145] Deleting tablet 5d167065029a46b791d413db8a2c4bcc
0112 20:51:36.833015 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.833143 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116345}
W20250112 20:51:36.833811 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5d167065029a46b791d413db8a2c4bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d167065029a46b791d413db8a2c4bcc already in progress: creating tablet
I20250112 20:51:36.835529 1163 tablet_bootstrap.cc:654] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.836138 1164 tablet_bootstrap.cc:654] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.838855 1186 ts_tablet_manager.cc:1918] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:36.839005 1186 log.cc:1198] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/2b8250621cbe45eba669b5ef14e672c2
I20250112 20:51:36.839411 1186 ts_tablet_manager.cc:1939] T 2b8250621cbe45eba669b5ef14e672c2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:36.840059 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 2b8250621cbe45eba669b5ef14e672c2 (table default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]) successfully deleted
I20250112 20:51:36.840508 1186 tablet_replica.cc:331] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:36.840821 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf66492d01a64eb1b842acc279b8ee09 consensus state has no leader
I20250112 20:51:36.841037 1186 raft_consensus.cc:2238] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:36.841253 1186 pending_rounds.cc:62] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:36.841297 1186 pending_rounds.cc:69] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585035994316800 op_type: NO_OP noop_request { }
I20250112 20:51:36.841396 1186 raft_consensus.cc:2883] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:36.841477 1186 raft_consensus.cc:2267] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.841796 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 491) took 1116 ms. Trace:
I20250112 20:51:36.841879 1184 rpcz_store.cc:276] 0112 20:51:35.726076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:35.726593 (+ 517us) service_pool.cc:225] Handling call
0112 20:51:36.841664 (+1115071us) ts_tablet_manager.cc:1145] Deleting tablet 921ae136d5e2438e950b96f156f23da3
0112 20:51:36.841672 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:36.841779 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114560}
I20250112 20:51:36.842108 1931 raft_consensus.cc:491] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.842227 1931 raft_consensus.cc:513] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:36.842358 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 921ae136d5e2438e950b96f156f23da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 921ae136d5e2438e950b96f156f23da3 already in progress: creating tablet
I20250112 20:51:36.842823 1931 leader_election.cc:290] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.842820 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9a9d57f9339042c0bdc18fd15adae7a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.843051 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4ef01b41b6140d6bdad07bc79dd7dc5 already in progress: creating tablet
W20250112 20:51:36.843355 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a9d57f9339042c0bdc18fd15adae7a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.843417 1184 tablet_replica.cc:331] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:36.843616 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 267ebb3c30194d9fa6a092d9bec4eadf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267ebb3c30194d9fa6a092d9bec4eadf already in progress: creating tablet
I20250112 20:51:36.843869 1184 raft_consensus.cc:2238] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.843889 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.844049 1184 raft_consensus.cc:2267] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.844071 910 raft_consensus.cc:2463] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:36.844427 710 leader_election.cc:304] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.844725 1931 raft_consensus.cc:2798] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.844817 1931 raft_consensus.cc:491] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.844889 1931 raft_consensus.cc:3054] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.846665 1184 ts_tablet_manager.cc:1905] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.847307 1931 raft_consensus.cc:513] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.847915 1931 leader_election.cc:290] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.848827 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.848986 1045 raft_consensus.cc:3054] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.849016 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:36.849164 910 raft_consensus.cc:3054] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.851130 1185 ts_tablet_manager.cc:1918] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:36.851217 1185 log.cc:1198] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f249d70b116443b7823e093efeda5cc3
I20250112 20:51:36.851403 1045 raft_consensus.cc:2463] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.851505 1185 ts_tablet_manager.cc:1939] T f249d70b116443b7823e093efeda5cc3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:36.851613 910 raft_consensus.cc:2463] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.852291 710 leader_election.cc:304] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.852687 1931 raft_consensus.cc:2798] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.852828 1931 raft_consensus.cc:695] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:36.853037 1931 consensus_queue.cc:237] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.854656 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f249d70b116443b7823e093efeda5cc3 (table default.Table_1f5073a477614ddfb8857091e89bc988 [id=ea1619ef75c0448a8834fbfde3bc4750]) successfully deleted
I20250112 20:51:36.855717 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.856226 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:36.856280 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 98547168378947df885c601ccdc85e57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98547168378947df885c601ccdc85e57 already in progress: creating tablet
W20250112 20:51:36.859733 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 65784493c6ef417993b1f6371cab47d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65784493c6ef417993b1f6371cab47d6 already in progress: creating tablet
W20250112 20:51:36.860710 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c already in progress: creating tablet
W20250112 20:51:36.861666 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b37b00c49f234dc89d63f644febc23fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37b00c49f234dc89d63f644febc23fb already in progress: creating tablet
I20250112 20:51:36.862907 1164 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.863021 1164 ts_tablet_manager.cc:1397] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250112 20:51:36.863663 1164 raft_consensus.cc:357] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.863832 1164 raft_consensus.cc:383] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.863875 1164 raft_consensus.cc:738] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.864621 1164 consensus_queue.cc:260] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.865229 1932 raft_consensus.cc:491] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.865356 1932 raft_consensus.cc:513] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.866045 1932 leader_election.cc:290] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.866525 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0a377dabb6b14129ba1e8519d731bafb" } modify_external_catalogs: true
I20250112 20:51:36.866743 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0a377dabb6b14129ba1e8519d731bafb" } modify_external_catalogs: true
I20250112 20:51:36.867487 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.867671 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:36.868817 710 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:36.869019 710 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.869087 710 leader_election.cc:304] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.869467 2059 raft_consensus.cc:2743] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.869614 1164 ts_tablet_manager.cc:1428] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:36.870833 1164 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:36.872852 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 7f807071e53a4f0e8629b1dfa6d26b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f807071e53a4f0e8629b1dfa6d26b9d already in progress: creating tablet
I20250112 20:51:36.875296 1164 tablet_bootstrap.cc:654] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:36.875995 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 already in progress: creating tablet
I20250112 20:51:36.876539 2133 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:36.875484 1185 tablet_replica.cc:331] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:36.876951 1185 raft_consensus.cc:2238] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.877099 1185 raft_consensus.cc:2267] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.877938 1184 ts_tablet_manager.cc:1918] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:36.878521 1185 ts_tablet_manager.cc:1905] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.881212 1184 log.cc:1198] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9c55b922c39545ac9c89115cd587ea8c
I20250112 20:51:36.878193 1163 tablet_bootstrap.cc:492] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:36.886858 1163 ts_tablet_manager.cc:1397] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.002s
I20250112 20:51:36.887696 1163 raft_consensus.cc:357] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.887974 1163 raft_consensus.cc:383] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.888056 1163 raft_consensus.cc:738] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.888291 1163 consensus_queue.cc:260] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.889622 1163 ts_tablet_manager.cc:1428] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:36.890527 1184 ts_tablet_manager.cc:1939] T 9c55b922c39545ac9c89115cd587ea8c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:36.891259 1186 ts_tablet_manager.cc:1905] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.892377 1821 raft_consensus.cc:491] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.892889 1821 raft_consensus.cc:513] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.893412 1821 leader_election.cc:290] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.892287 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda44c48bf9746b59ccdc4d39a41ea91" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.893651 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9c55b922c39545ac9c89115cd587ea8c (table default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]) successfully deleted
I20250112 20:51:36.893671 1048 raft_consensus.cc:2371] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1. Re-sending same reply.
I20250112 20:51:36.894150 1184 tablet_replica.cc:331] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:36.894126 1931 raft_consensus.cc:491] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.894308 1184 raft_consensus.cc:2238] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:36.894263 1931 raft_consensus.cc:513] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.894433 1184 raft_consensus.cc:2267] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:36.894398 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eb0f79df5444f8a8360921d5f2b9ee" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.894604 770 raft_consensus.cc:2463] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:36.894906 1184 ts_tablet_manager.cc:1905] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:36.894910 1931 leader_election.cc:290] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:36.895015 983 leader_election.cc:343] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:36.895737 983 leader_election.cc:304] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:36.896190 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bf66492d01a64eb1b842acc279b8ee09 (table default.Table_ee975b071729460b863016012a403cb4 [id=e1d57fdfdc434d36bb69c54d5c974f41], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bf66492d01a64eb1b842acc279b8ee09 consensus state has no leader
I20250112 20:51:36.896899 1821 raft_consensus.cc:2798] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.896975 1821 raft_consensus.cc:491] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.897033 1821 raft_consensus.cc:3054] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.897142 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8c9a7a06fb384d4aa61879e57247d36e" } modify_external_catalogs: true
I20250112 20:51:36.897336 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8c9a7a06fb384d4aa61879e57247d36e" } modify_external_catalogs: true
I20250112 20:51:36.899842 1164 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:36.899967 1164 ts_tablet_manager.cc:1397] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250112 20:51:36.900251 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1454445414a4f2882deedb81bad67bf" } modify_external_catalogs: true
I20250112 20:51:36.900420 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a1454445414a4f2882deedb81bad67bf" } modify_external_catalogs: true
I20250112 20:51:36.900861 1821 raft_consensus.cc:513] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.901227 1164 raft_consensus.cc:357] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.901466 1164 raft_consensus.cc:383] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.901750 1163 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:36.903308 1163 tablet_bootstrap.cc:654] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.903405 1164 raft_consensus.cc:738] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:36.904549 983 leader_election.cc:343] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:36.904461 1164 consensus_queue.cc:260] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.905037 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eb0f79df5444f8a8360921d5f2b9ee" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.905141 1164 ts_tablet_manager.cc:1428] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:36.905211 770 raft_consensus.cc:3054] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.907814 770 raft_consensus.cc:2463] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:36.907831 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e07c52b8ba4040188dcdd21d2ffa00a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:36.908373 983 leader_election.cc:304] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:36.908833 2140 raft_consensus.cc:2798] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.905284 1821 leader_election.cc:290] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:36.909286 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b30a34c1abcb4f72a7f4f8a8f5be3720 consensus state has no leader
W20250112 20:51:36.911164 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e07c52b8ba4040188dcdd21d2ffa00a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:36.917896 1178 tablet_bootstrap.cc:492] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:36.918023 1178 ts_tablet_manager.cc:1397] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.161s user 0.011s sys 0.015s
I20250112 20:51:36.918799 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.918984 1048 raft_consensus.cc:2463] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:36.918885 1178 raft_consensus.cc:357] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.919081 1178 raft_consensus.cc:383] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:36.919154 1178 raft_consensus.cc:738] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:36.919317 710 leader_election.cc:304] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.919373 1178 consensus_queue.cc:260] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.920179 1178 ts_tablet_manager.cc:1428] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:36.920379 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:36.921011 1931 raft_consensus.cc:2798] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.921141 911 raft_consensus.cc:2463] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:36.921156 1931 raft_consensus.cc:491] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.921347 1931 raft_consensus.cc:3054] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.922392 1164 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:36.922834 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 68b4b312c558437892388344767f0ed4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:36.923854 1931 raft_consensus.cc:513] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.923992 1164 tablet_bootstrap.cc:654] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.924408 1931 leader_election.cc:290] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:36.924994 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:36.925143 910 raft_consensus.cc:3054] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.926293 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f0daaeb0bf6046da8de969b1f4c71c84 with delete_type TABLET_DATA_DELETED (replaced by tablet ab9be1f065ec4605a0a3fe69770793d0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.929296 910 raft_consensus.cc:2463] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.929590 2152 tablet.cc:1721] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:36.929848 710 leader_election.cc:304] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:36.930938 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0daaeb0bf6046da8de969b1f4c71c84" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:36.931174 1048 raft_consensus.cc:3054] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.932204 2059 raft_consensus.cc:2798] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:36.932315 2059 raft_consensus.cc:695] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:36.932153 450 catalog_manager.cc:5526] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:36.932533 2059 consensus_queue.cc:237] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.939221 737 tablet_service.cc:1467] Processing CreateTablet for tablet dfc63eff44414906a027a8208b5cc5a9 (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.939308 741 tablet_service.cc:1467] Processing CreateTablet for tablet 960063ff575f4a9fa9298d20832b289d (DEFAULT_TABLE table=default.Table_3e98d1a51a4e4971afa54b9847d7888b [id=9d97c0dc685b4fc1b49a1a43e1fa19a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.939522 1048 raft_consensus.cc:2463] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:36.939585 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc63eff44414906a027a8208b5cc5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.939841 754 tablet_service.cc:1467] Processing CreateTablet for tablet d76c0b26b634467087ed94fb6d02f075 (DEFAULT_TABLE table=default.Table_72a560b4d6274b2c9fc00315582a7894 [id=d9b0e02cec0f4b34abc5d5ba6692c907]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.940006 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc719cd84124082b782016011e37c8b (DEFAULT_TABLE table=default.Table_dcdefc016f7546e39891ebe15594cba1 [id=9ef53ea9f8ce43978f93e6234c274905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.940040 2140 raft_consensus.cc:695] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:36.940140 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76c0b26b634467087ed94fb6d02f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.940284 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc719cd84124082b782016011e37c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.940287 2140 consensus_queue.cc:237] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.940569 739 tablet_service.cc:1467] Processing CreateTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 (DEFAULT_TABLE table=default.Table_1175f36a82364f40bf29fd45784c07a0 [id=890c5f85f2b74c388afd28c28624a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.940886 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6158b97ccf0a42fdb1a510c5d1e80213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.944092 2108 raft_consensus.cc:491] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.944211 2108 raft_consensus.cc:513] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.944363 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1b866eb75ab4415ca97ffadfbdf1278b (DEFAULT_TABLE table=default.Table_57a800cd62ef4adea24037854c5f479f [id=b597b65188ae4ec7a99f1d095b5969d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.944594 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960063ff575f4a9fa9298d20832b289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.944814 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b866eb75ab4415ca97ffadfbdf1278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.944769 2108 leader_election.cc:290] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.948657 893 tablet_service.cc:1467] Processing CreateTablet for tablet 960063ff575f4a9fa9298d20832b289d (DEFAULT_TABLE table=default.Table_3e98d1a51a4e4971afa54b9847d7888b [id=9d97c0dc685b4fc1b49a1a43e1fa19a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.949091 888 tablet_service.cc:1467] Processing CreateTablet for tablet d76c0b26b634467087ed94fb6d02f075 (DEFAULT_TABLE table=default.Table_72a560b4d6274b2c9fc00315582a7894 [id=d9b0e02cec0f4b34abc5d5ba6692c907]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.949234 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 (DEFAULT_TABLE table=default.Table_1175f36a82364f40bf29fd45784c07a0 [id=890c5f85f2b74c388afd28c28624a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.949417 877 tablet_service.cc:1467] Processing CreateTablet for tablet dfc63eff44414906a027a8208b5cc5a9 (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.949570 2108 raft_consensus.cc:491] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:36.949597 878 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc719cd84124082b782016011e37c8b (DEFAULT_TABLE table=default.Table_dcdefc016f7546e39891ebe15594cba1 [id=9ef53ea9f8ce43978f93e6234c274905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.949645 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66c55c29de7458bb8ea69b826c567d7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.949684 2108 raft_consensus.cc:513] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.949894 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1b866eb75ab4415ca97ffadfbdf1278b (DEFAULT_TABLE table=default.Table_57a800cd62ef4adea24037854c5f479f [id=b597b65188ae4ec7a99f1d095b5969d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.950192 2108 leader_election.cc:290] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.950378 882 tablet_service.cc:1467] Processing CreateTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 (DEFAULT_TABLE table=default.Table_2f7947f8e7af4b3799d132d1f226f57a [id=1d4bedab84374d70ab926dd9381fdd23]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:36.951076 847 leader_election.cc:343] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.952016 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66c55c29de7458bb8ea69b826c567d7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.952121 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:36.952210 770 raft_consensus.cc:2463] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:36.954559 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 33a90f801d854d4d8f7f09112e2d2929
I20250112 20:51:36.956136 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:36.938884 753 tablet_service.cc:1467] Processing CreateTablet for tablet 88463f03609e43f6b39ecc555ac353fa (DEFAULT_TABLE table=default.Table_2a1cc8e9bac14fc7ab9057e1dd447fb4 [id=eb562f22ab0243fea3163472c113c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.957865 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 88463f03609e43f6b39ecc555ac353fa (DEFAULT_TABLE table=default.Table_2a1cc8e9bac14fc7ab9057e1dd447fb4 [id=eb562f22ab0243fea3163472c113c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.958194 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88463f03609e43f6b39ecc555ac353fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.958135 880 tablet_service.cc:1467] Processing CreateTablet for tablet 88463f03609e43f6b39ecc555ac353fa (DEFAULT_TABLE table=default.Table_2a1cc8e9bac14fc7ab9057e1dd447fb4 [id=eb562f22ab0243fea3163472c113c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.958819 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88463f03609e43f6b39ecc555ac353fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.961265 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 960063ff575f4a9fa9298d20832b289d (DEFAULT_TABLE table=default.Table_3e98d1a51a4e4971afa54b9847d7888b [id=9d97c0dc685b4fc1b49a1a43e1fa19a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.961664 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960063ff575f4a9fa9298d20832b289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.965855 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d76c0b26b634467087ed94fb6d02f075 (DEFAULT_TABLE table=default.Table_72a560b4d6274b2c9fc00315582a7894 [id=d9b0e02cec0f4b34abc5d5ba6692c907]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.966179 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76c0b26b634467087ed94fb6d02f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.969178 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 (DEFAULT_TABLE table=default.Table_1175f36a82364f40bf29fd45784c07a0 [id=890c5f85f2b74c388afd28c28624a4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.969466 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6158b97ccf0a42fdb1a510c5d1e80213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.969523 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:36.971356 2157 tablet.cc:1721] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:36.972992 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc719cd84124082b782016011e37c8b (DEFAULT_TABLE table=default.Table_dcdefc016f7546e39891ebe15594cba1 [id=9ef53ea9f8ce43978f93e6234c274905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.973291 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc719cd84124082b782016011e37c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.976975 1022 tablet_service.cc:1467] Processing CreateTablet for tablet dfc63eff44414906a027a8208b5cc5a9 (DEFAULT_TABLE table=default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.977260 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc63eff44414906a027a8208b5cc5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.969698 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ad7cf1e0c74265a4d21931279f2318" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:36.979430 1048 raft_consensus.cc:2388] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:36.920912 1178 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:36.981015 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 1b866eb75ab4415ca97ffadfbdf1278b (DEFAULT_TABLE table=default.Table_57a800cd62ef4adea24037854c5f479f [id=b597b65188ae4ec7a99f1d095b5969d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.981304 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b866eb75ab4415ca97ffadfbdf1278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.982364 1178 tablet_bootstrap.cc:654] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:36.984434 847 leader_election.cc:304] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:36.984835 847 leader_election.cc:304] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:36.985332 2076 raft_consensus.cc:2798] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:36.985361 1942 raft_consensus.cc:3054] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.985409 2076 raft_consensus.cc:491] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:36.985399 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 (DEFAULT_TABLE table=default.Table_2f7947f8e7af4b3799d132d1f226f57a [id=1d4bedab84374d70ab926dd9381fdd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:36.985469 2076 raft_consensus.cc:3054] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.985704 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cfb20cbf384b559bea4528e5f37bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:36.987910 2076 raft_consensus.cc:513] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:36.988023 1942 raft_consensus.cc:2743] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:36.988494 2076 leader_election.cc:290] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:36.988927 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a90f801d854d4d8f7f09112e2d2929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:36.990566 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66c55c29de7458bb8ea69b826c567d7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:36.991233 847 leader_election.cc:343] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:36.993155 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_10962594cedf43f6a5533239b618afa4" } new_table_name: "default.Table_4372511dc98c437da09ce4a0aab012f7" modify_external_catalogs: true
I20250112 20:51:36.993701 2131 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:36.994292 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66c55c29de7458bb8ea69b826c567d7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:36.994498 775 raft_consensus.cc:3054] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:36.994455 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_47f89b433c7e4b11a6e685b3c702c3d1" } new_table_name: "default.Table_830127232d00465b966ceb3ecce56921" modify_external_catalogs: true
I20250112 20:51:36.999383 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c186d9fb7acb41a8950a79e3d23262a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:36.999933 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c186d9fb7acb41a8950a79e3d23262a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.004096 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49a14f14146009126769791dcd780 (DEFAULT_TABLE table=default.Table_669829b96b0a4f1fab410ba1e9b1e5d6 [id=f09e01680bc748a096b5651e2fe6aaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.004554 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df49a14f14146009126769791dcd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.005311 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9 (DEFAULT_TABLE table=default.Table_a7e6ee6b896447d3aa166e1d4cfb2d79 [id=88ff328a835746e894371f9f98161b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.005511 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49a14f14146009126769791dcd780 (DEFAULT_TABLE table=default.Table_669829b96b0a4f1fab410ba1e9b1e5d6 [id=f09e01680bc748a096b5651e2fe6aaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.005693 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.006034 739 tablet_service.cc:1467] Processing CreateTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 (DEFAULT_TABLE table=default.Table_2f7947f8e7af4b3799d132d1f226f57a [id=1d4bedab84374d70ab926dd9381fdd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.006305 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cfb20cbf384b559bea4528e5f37bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.009301 747 tablet_service.cc:1467] Processing CreateTablet for tablet 69d57977510c4422ad625a7045ede7e4 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.009423 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df49a14f14146009126769791dcd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.009497 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9 (DEFAULT_TABLE table=default.Table_a7e6ee6b896447d3aa166e1d4cfb2d79 [id=88ff328a835746e894371f9f98161b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.009656 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d57977510c4422ad625a7045ede7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.009794 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.005276 746 tablet_service.cc:1467] Processing CreateTablet for tablet 51d2a44677aa492782c7c805300ed6a5 (DEFAULT_TABLE table=default.Table_bf1f1d5c2be64a72af1c3c8396cb7180 [id=973204a25fc54cf4a7dc11c7644d17b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.013027 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 53fcb48fe5314957ae807902274ef596 (DEFAULT_TABLE table=default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.013145 874 tablet_service.cc:1467] Processing CreateTablet for tablet 0df49a14f14146009126769791dcd780 (DEFAULT_TABLE table=default.Table_669829b96b0a4f1fab410ba1e9b1e5d6 [id=f09e01680bc748a096b5651e2fe6aaba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.013244 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2a44677aa492782c7c805300ed6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.013340 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fcb48fe5314957ae807902274ef596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.013659 886 tablet_service.cc:1467] Processing CreateTablet for tablet 53fcb48fe5314957ae807902274ef596 (DEFAULT_TABLE table=default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.014122 889 tablet_service.cc:1467] Processing CreateTablet for tablet 51d2a44677aa492782c7c805300ed6a5 (DEFAULT_TABLE table=default.Table_bf1f1d5c2be64a72af1c3c8396cb7180 [id=973204a25fc54cf4a7dc11c7644d17b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.014566 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9 (DEFAULT_TABLE table=default.Table_a7e6ee6b896447d3aa166e1d4cfb2d79 [id=88ff328a835746e894371f9f98161b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.015098 879 tablet_service.cc:1467] Processing CreateTablet for tablet 69d57977510c4422ad625a7045ede7e4 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.015771 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 69d57977510c4422ad625a7045ede7e4 (DEFAULT_TABLE table=default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.016131 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d57977510c4422ad625a7045ede7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.019097 2133 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.023702 1184 ts_tablet_manager.cc:1918] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.023926 1184 log.cc:1198] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/81f0875e271d4aaf9dc5da6aa8113ed0
I20250112 20:51:37.024231 1184 ts_tablet_manager.cc:1939] T 81f0875e271d4aaf9dc5da6aa8113ed0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.029057 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 51d2a44677aa492782c7c805300ed6a5 (DEFAULT_TABLE table=default.Table_bf1f1d5c2be64a72af1c3c8396cb7180 [id=973204a25fc54cf4a7dc11c7644d17b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.031052 1185 ts_tablet_manager.cc:1918] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.031391 1185 log.cc:1198] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e44422f75a744d078cdf4f0997242e22
I20250112 20:51:37.031738 1185 ts_tablet_manager.cc:1939] T e44422f75a744d078cdf4f0997242e22 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.033612 1164 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.033737 1164 ts_tablet_manager.cc:1397] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20250112 20:51:37.034286 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2a44677aa492782c7c805300ed6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.038259 1163 tablet_bootstrap.cc:492] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.038388 1163 ts_tablet_manager.cc:1397] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.002s
I20250112 20:51:37.038466 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960063ff575f4a9fa9298d20832b289d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.039184 1163 raft_consensus.cc:357] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.039373 1163 raft_consensus.cc:383] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.039415 1163 raft_consensus.cc:738] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.039566 1163 consensus_queue.cc:260] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.040010 1163 ts_tablet_manager.cc:1428] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:37.040562 1164 raft_consensus.cc:357] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.040724 1164 raft_consensus.cc:383] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.040797 1164 raft_consensus.cc:738] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.041222 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6158b97ccf0a42fdb1a510c5d1e80213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.043780 1186 ts_tablet_manager.cc:1918] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.043875 1186 log.cc:1198] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/54de54eabe5946058d1940b79f5fb832
I20250112 20:51:37.044200 1186 ts_tablet_manager.cc:1939] T 54de54eabe5946058d1940b79f5fb832 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.044979 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2a44677aa492782c7c805300ed6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.044945 1164 consensus_queue.cc:260] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.046335 1163 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.048334 775 raft_consensus.cc:2463] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:37.048918 1163 tablet_bootstrap.cc:654] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.048981 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dc719cd84124082b782016011e37c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.049424 1178 tablet_bootstrap.cc:492] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.049535 1178 ts_tablet_manager.cc:1397] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.000s
I20250112 20:51:37.050204 1178 raft_consensus.cc:357] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.050359 1178 raft_consensus.cc:383] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.050397 1178 raft_consensus.cc:738] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.050511 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e44422f75a744d078cdf4f0997242e22 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
I20250112 20:51:37.050580 1178 consensus_queue.cc:260] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.052960 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1902860ff1ff46928dbbb6d8ebf2e7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.056594 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc63eff44414906a027a8208b5cc5a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:37.056967 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 64cd83b199bb4da28d39da449945c389 because the tablet was not found. No further retry: Not found: Tablet not found: 64cd83b199bb4da28d39da449945c389
I20250112 20:51:37.057250 1185 tablet_replica.cc:331] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.057382 1164 ts_tablet_manager.cc:1428] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250112 20:51:37.058689 1185 raft_consensus.cc:2238] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.058847 1185 raft_consensus.cc:2267] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.058935 753 tablet_service.cc:1467] Processing CreateTablet for tablet 53fcb48fe5314957ae807902274ef596 (DEFAULT_TABLE table=default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.059271 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fcb48fe5314957ae807902274ef596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.059711 1178 ts_tablet_manager.cc:1428] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:37.059334 1185 ts_tablet_manager.cc:1905] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.060168 1178 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.060588 1164 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.061009 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 54de54eabe5946058d1940b79f5fb832 (table default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]) successfully deleted
I20250112 20:51:37.061427 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b866eb75ab4415ca97ffadfbdf1278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.063207 1178 tablet_bootstrap.cc:654] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.061564 1186 tablet_replica.cc:331] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.064324 1186 raft_consensus.cc:2238] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.064455 1186 raft_consensus.cc:2267] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.065034 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88463f03609e43f6b39ecc555ac353fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.067071 1186 ts_tablet_manager.cc:1905] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.068096 1164 tablet_bootstrap.cc:654] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.068584 1184 tablet_replica.cc:331] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.069002 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76c0b26b634467087ed94fb6d02f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.069350 1184 raft_consensus.cc:2238] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.069484 1184 raft_consensus.cc:2267] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.069895 1184 ts_tablet_manager.cc:1905] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.070550 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cfb20cbf384b559bea4528e5f37bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.072507 1186 ts_tablet_manager.cc:1918] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.072587 1186 log.cc:1198] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f4cae067df1a40098f81cfbd17224c75
I20250112 20:51:37.072839 1186 ts_tablet_manager.cc:1939] T f4cae067df1a40098f81cfbd17224c75 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.073454 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53fcb48fe5314957ae807902274ef596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.073604 1186 tablet_replica.cc:331] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.073694 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f4cae067df1a40098f81cfbd17224c75 (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
I20250112 20:51:37.073770 1186 raft_consensus.cc:2238] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.073889 1186 raft_consensus.cc:2267] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.074313 1186 ts_tablet_manager.cc:1905] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.076961 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df49a14f14146009126769791dcd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.080981 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d57977510c4422ad625a7045ede7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.100504 1186 ts_tablet_manager.cc:1918] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.100608 1186 log.cc:1198] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/eda44c48bf9746b59ccdc4d39a41ea91
I20250112 20:51:37.100883 1186 ts_tablet_manager.cc:1939] T eda44c48bf9746b59ccdc4d39a41ea91 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.102079 1185 ts_tablet_manager.cc:1918] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.102197 1185 log.cc:1198] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:37.102315 847 leader_election.cc:304] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:37.102566 1982 raft_consensus.cc:2798] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.102664 1982 raft_consensus.cc:695] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:37.102886 1982 consensus_queue.cc:237] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.105357 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet eda44c48bf9746b59ccdc4d39a41ea91 (table default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]) successfully deleted
I20250112 20:51:37.109277 2157 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.102502 1185 ts_tablet_manager.cc:1939] T 42eb0f79df5444f8a8360921d5f2b9ee P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.124135 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 42eb0f79df5444f8a8360921d5f2b9ee (table default.Table_43f4ea15650c4619afdabdb3c169df86 [id=dd64a2679c0849279e9dae49f268056b]) successfully deleted
I20250112 20:51:37.126543 1184 ts_tablet_manager.cc:1918] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.126909 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 81f0875e271d4aaf9dc5da6aa8113ed0 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
W20250112 20:51:37.131595 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 691e2c3f72ab4f439678d166b9b31661 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e2c3f72ab4f439678d166b9b31661 already in progress: creating tablet
W20250112 20:51:37.132180 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 104d6b4effb84314a6b2b9db883b4366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104d6b4effb84314a6b2b9db883b4366 already in progress: creating tablet
I20250112 20:51:37.135624 1186 tablet_replica.cc:331] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.135872 1186 raft_consensus.cc:2238] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.136034 1186 raft_consensus.cc:2267] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.136312 1185 tablet_replica.cc:331] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.138309 1186 ts_tablet_manager.cc:1905] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.138684 1185 raft_consensus.cc:2238] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.139396 1185 pending_rounds.cc:62] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:37.139945 1185 pending_rounds.cc:69] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585035073339392 op_type: NO_OP noop_request { }
I20250112 20:51:37.140339 1185 raft_consensus.cc:2883] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.140663 1185 raft_consensus.cc:2267] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.141265 1184 log.cc:1198] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:37.141654 1185 ts_tablet_manager.cc:1905] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.142201 1184 ts_tablet_manager.cc:1939] T 6fe28f23448c47d3979f1982f939e39c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.146248 1184 tablet_replica.cc:331] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.146497 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 6fe28f23448c47d3979f1982f939e39c (table default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]) successfully deleted
W20250112 20:51:37.146979 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c already in progress: creating tablet
W20250112 20:51:37.147343 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 98547168378947df885c601ccdc85e57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98547168378947df885c601ccdc85e57 already in progress: creating tablet
W20250112 20:51:37.147670 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 65784493c6ef417993b1f6371cab47d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65784493c6ef417993b1f6371cab47d6 already in progress: creating tablet
I20250112 20:51:37.147753 1185 ts_tablet_manager.cc:1918] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.147868 1185 log.cc:1198] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b4ad7cf1e0c74265a4d21931279f2318
W20250112 20:51:37.148138 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b37b00c49f234dc89d63f644febc23fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37b00c49f234dc89d63f644febc23fb already in progress: creating tablet
I20250112 20:51:37.148172 1185 ts_tablet_manager.cc:1939] T b4ad7cf1e0c74265a4d21931279f2318 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:37.149060 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7f807071e53a4f0e8629b1dfa6d26b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f807071e53a4f0e8629b1dfa6d26b9d already in progress: creating tablet
I20250112 20:51:37.153817 1184 raft_consensus.cc:2238] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.154026 1184 raft_consensus.cc:2267] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.157320 1184 ts_tablet_manager.cc:1905] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.157894 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b4ad7cf1e0c74265a4d21931279f2318 (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
I20250112 20:51:37.154352 1163 tablet_bootstrap.cc:492] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.158172 1163 ts_tablet_manager.cc:1397] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.008s
I20250112 20:51:37.158213 1185 tablet_replica.cc:331] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.158389 1185 raft_consensus.cc:2238] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.158520 1185 raft_consensus.cc:2267] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.158937 1185 ts_tablet_manager.cc:1905] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.159193 1163 raft_consensus.cc:357] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.159435 1163 raft_consensus.cc:383] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:37.159507 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250112 20:51:37.159540 1163 raft_consensus.cc:738] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.159924 1163 consensus_queue.cc:260] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.161419 1163 ts_tablet_manager.cc:1428] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:37.163916 2131 tablet.cc:1721] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.162768 1163 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.170369 1178 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.170507 1178 ts_tablet_manager.cc:1397] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.110s user 0.018s sys 0.013s
I20250112 20:51:37.171308 1178 raft_consensus.cc:357] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.171496 1178 raft_consensus.cc:383] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.171558 1178 raft_consensus.cc:738] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.171829 1164 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.171761 1178 consensus_queue.cc:260] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.172242 1178 ts_tablet_manager.cc:1428] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:37.172564 1164 ts_tablet_manager.cc:1397] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.000s
I20250112 20:51:37.172986 1178 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.174619 1178 tablet_bootstrap.cc:654] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.174836 1164 raft_consensus.cc:357] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.175384 1164 raft_consensus.cc:383] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.175649 1164 raft_consensus.cc:738] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.175951 1164 consensus_queue.cc:260] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.176493 1164 ts_tablet_manager.cc:1428] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:37.177328 1164 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:37.180231 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer 0f5c09e899184de6b770dca873501062 (127.0.104.60:45153): Couldn't send request to peer 0f5c09e899184de6b770dca873501062. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250112 20:51:37.184427 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_e2d706e7614440a1905f8790f025ac29 [id=75ce02402d924e7f8153b6348a26af9e] alter complete (version 2)
I20250112 20:51:37.186477 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eb0f79df5444f8a8360921d5f2b9ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.187184 1185 ts_tablet_manager.cc:1918] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.187292 1185 log.cc:1198] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/54de54eabe5946058d1940b79f5fb832
I20250112 20:51:37.187573 1185 ts_tablet_manager.cc:1939] T 54de54eabe5946058d1940b79f5fb832 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.188625 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 54de54eabe5946058d1940b79f5fb832 (table default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]) successfully deleted
I20250112 20:51:37.189703 1163 tablet_bootstrap.cc:654] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.194691 1164 tablet_bootstrap.cc:654] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.196919 1185 tablet_replica.cc:331] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.197113 1185 raft_consensus.cc:2238] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.197324 1185 pending_rounds.cc:62] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:37.197367 1185 pending_rounds.cc:69] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585037037936640 op_type: NO_OP noop_request { }
I20250112 20:51:37.197463 1185 raft_consensus.cc:2883] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.197536 1185 raft_consensus.cc:2267] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.198354 1185 ts_tablet_manager.cc:1905] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.201273 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_037b9792e05d4bd08c1ae042e95cba5f" } modify_external_catalogs: true
I20250112 20:51:37.201519 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_037b9792e05d4bd08c1ae042e95cba5f" } modify_external_catalogs: true
I20250112 20:51:37.202700 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1ca8fc4a45fe448b919ac001a7980deb
I20250112 20:51:37.203372 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5c98d71957e443d48bf632f3fe662c5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.203894 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c98d71957e443d48bf632f3fe662c5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.204321 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 68b4b312c558437892388344767f0ed4
I20250112 20:51:37.204753 446 catalog_manager.cc:6596] ReplaceTablet: tablet 68b4b312c558437892388344767f0ed4 deleted and replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4
I20250112 20:51:37.204989 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f875f075e98e45499291b519caf03184 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.205395 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.205616 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.206187 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 95cbe4933f2b49ebbdb6eb7f0f8db088
I20250112 20:51:37.206594 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b4b312c558437892388344767f0ed4 with delete_type TABLET_DATA_DELETED (replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.206954 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b4b312c558437892388344767f0ed4 with delete_type TABLET_DATA_DELETED (replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.207592 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.207989 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.208380 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.208441 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 95d8e68746cf444ebac5e22728ab6f34
I20250112 20:51:37.208557 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b4b312c558437892388344767f0ed4 with delete_type TABLET_DATA_DELETED (replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.208703 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.209062 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.209127 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_06d8dc6fe3af4be68fee565a90654657" } new_table_name: "default.Table_093ad5dba68a4f0b9d96c37ac3888a83" modify_external_catalogs: true
I20250112 20:51:37.209717 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.210053 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:37.210317 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06253bb13a9541d8b408d1bf73e469dd consensus state has no leader
I20250112 20:51:37.210681 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_93052f6902b2404d9a83652fca013746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.211170 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93052f6902b2404d9a83652fca013746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.212477 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4b1f4ea776424da59c8ec69715c5337d" } new_table_name: "default.Table_285cfdca273f457c98ac4d491dfe84af" modify_external_catalogs: true
I20250112 20:51:37.217180 1178 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.217317 1178 ts_tablet_manager.cc:1397] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.044s user 0.023s sys 0.011s
I20250112 20:51:37.218113 1178 raft_consensus.cc:357] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.218312 1178 raft_consensus.cc:383] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.218380 1178 raft_consensus.cc:738] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.218583 1178 consensus_queue.cc:260] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:37.221781 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60eefdbabc174cd6a67fd12ea7d95071 consensus state has no leader
I20250112 20:51:37.221788 1184 ts_tablet_manager.cc:1918] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.221875 1178 ts_tablet_manager.cc:1428] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:37.221900 1184 log.cc:1198] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f0daaeb0bf6046da8de969b1f4c71c84
I20250112 20:51:37.222329 1178 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.222214 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_91d16ee866374194947b9edddcf2e992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:37.222467 1184 ts_tablet_manager.cc:1939] T f0daaeb0bf6046da8de969b1f4c71c84 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:37.222669 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91d16ee866374194947b9edddcf2e992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.223420 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f0daaeb0bf6046da8de969b1f4c71c84 (table default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]) successfully deleted
I20250112 20:51:37.224637 1178 tablet_bootstrap.cc:654] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:37.224957 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 already in progress: creating tablet
I20250112 20:51:37.225421 1184 tablet_replica.cc:331] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.225592 1184 raft_consensus.cc:2238] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.225721 1184 raft_consensus.cc:2267] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.226182 1184 ts_tablet_manager.cc:1905] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.238270 1178 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.238413 1178 ts_tablet_manager.cc:1397] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20250112 20:51:37.239171 1178 raft_consensus.cc:357] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.239378 1178 raft_consensus.cc:383] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.239439 1178 raft_consensus.cc:738] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.239634 1178 consensus_queue.cc:260] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.242808 1178 ts_tablet_manager.cc:1428] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:37.243081 1178 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.244473 1184 ts_tablet_manager.cc:1918] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.244581 1184 log.cc:1198] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e44422f75a744d078cdf4f0997242e22
I20250112 20:51:37.244765 1178 tablet_bootstrap.cc:654] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.248335 2140 raft_consensus.cc:491] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.248472 2140 raft_consensus.cc:513] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.249130 2140 leader_election.cc:290] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:37.249456 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 640dc9187c2243248ef73bac6616c126 consensus state has no leader
I20250112 20:51:37.249924 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_be64887db24d47379842a579e8c4d332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.250344 983 leader_election.cc:343] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 8f636df32bd34438bb54989736eaa431
W20250112 20:51:37.250522 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be64887db24d47379842a579e8c4d332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:37.250689 983 leader_election.cc:343] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 8f636df32bd34438bb54989736eaa431
I20250112 20:51:37.250813 983 leader_election.cc:304] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:37.251230 2140 raft_consensus.cc:2743] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.251332 1178 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.249136 1184 ts_tablet_manager.cc:1939] T e44422f75a744d078cdf4f0997242e22 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.251439 1178 ts_tablet_manager.cc:1397] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250112 20:51:37.251847 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f875f075e98e45499291b519caf03184
I20250112 20:51:37.252346 446 catalog_manager.cc:6596] ReplaceTablet: tablet f875f075e98e45499291b519caf03184 deleted and replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e
I20250112 20:51:37.252562 446 master_service.cc:946] ReplaceTablet: received request to replace tablet e3648997eb1241edaf56c825bf76b240 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.252663 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.252923 883 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.252722 753 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.253173 1178 raft_consensus.cc:357] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.253334 1178 raft_consensus.cc:383] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.253373 1178 raft_consensus.cc:738] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.253554 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e44422f75a744d078cdf4f0997242e22 (table default.Table_ceaaef5837a2461f90a9ca600452abf6 [id=9d2d3ff310164745b68eda5bcc582fd7]) successfully deleted
I20250112 20:51:37.253561 1178 consensus_queue.cc:260] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:37.253952 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a87da1ea4054ffc81b9c1c792f0cc5d consensus state has no leader
I20250112 20:51:37.254053 1178 ts_tablet_manager.cc:1428] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:37.254350 1178 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.254305 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f4d062a92b0d473e8d60d7224cbc4f9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.254812 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4d062a92b0d473e8d60d7224cbc4f9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.256117 1178 tablet_bootstrap.cc:654] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.256769 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2a7113eb6fe54951860e36d9f3385a5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.257344 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a7113eb6fe54951860e36d9f3385a5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.258723 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4e1968191ac14099b6035ac487e3e6bb
I20250112 20:51:37.259445 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dc0bcf302ab44599b97fbba9f0301582" } new_table_name: "default.Table_9d078f3dac874426bfcdd9504733991c" modify_external_catalogs: true
I20250112 20:51:37.260212 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1968191ac14099b6035ac487e3e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.260459 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1968191ac14099b6035ac487e3e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.261083 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d030f9e0a0474434bb26dd47e6a66afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.261515 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d030f9e0a0474434bb26dd47e6a66afe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.253830 1184 tablet_replica.cc:331] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.261034 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1968191ac14099b6035ac487e3e6bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.263664 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c6d5d425e6a493daa46900cbd8db166 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.263687 1184 raft_consensus.cc:2238] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.263844 1184 raft_consensus.cc:2267] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.264197 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_31360e3161b04902a56502dbe4608f86" } modify_external_catalogs: true
I20250112 20:51:37.264384 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_31360e3161b04902a56502dbe4608f86" } modify_external_catalogs: true
I20250112 20:51:37.264513 1184 ts_tablet_manager.cc:1905] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.271011 1184 ts_tablet_manager.cc:1918] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.271116 1184 log.cc:1198] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4688dd6bbfc046b7bb140206cae9f104
I20250112 20:51:37.271445 1184 ts_tablet_manager.cc:1939] T 4688dd6bbfc046b7bb140206cae9f104 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.272423 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4688dd6bbfc046b7bb140206cae9f104 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]) successfully deleted
I20250112 20:51:37.273339 1178 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.273473 1178 ts_tablet_manager.cc:1397] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20250112 20:51:37.274259 1178 raft_consensus.cc:357] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.274443 1178 raft_consensus.cc:383] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.274497 1178 raft_consensus.cc:738] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.274721 1178 consensus_queue.cc:260] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.275357 1178 ts_tablet_manager.cc:1428] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.276473 1178 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.278211 1178 tablet_bootstrap.cc:654] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.278949 1184 tablet_replica.cc:331] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.279131 1184 raft_consensus.cc:2238] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.279433 1184 raft_consensus.cc:2267] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.279932 1184 ts_tablet_manager.cc:1905] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.283314 2059 raft_consensus.cc:491] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.283480 2059 raft_consensus.cc:513] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.284080 2059 leader_election.cc:290] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.286278 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:37.286561 1184 ts_tablet_manager.cc:1918] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.286507 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.286684 1184 log.cc:1198] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b4ad7cf1e0c74265a4d21931279f2318
I20250112 20:51:37.286945 1184 ts_tablet_manager.cc:1939] T b4ad7cf1e0c74265a4d21931279f2318 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:37.287009 710 leader_election.cc:343] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:37.287240 710 leader_election.cc:343] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.287314 710 leader_election.cc:304] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:37.287647 2059 raft_consensus.cc:2743] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.287955 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b4ad7cf1e0c74265a4d21931279f2318 (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
I20250112 20:51:37.288230 1184 tablet_replica.cc:331] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.288434 1184 raft_consensus.cc:2238] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.288555 1184 raft_consensus.cc:2267] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:37.288645 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 691e2c3f72ab4f439678d166b9b31661 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e2c3f72ab4f439678d166b9b31661 already in progress: creating tablet
W20250112 20:51:37.288962 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 104d6b4effb84314a6b2b9db883b4366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104d6b4effb84314a6b2b9db883b4366 already in progress: creating tablet
I20250112 20:51:37.289063 1184 ts_tablet_manager.cc:1905] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:37.289234 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3b532068d2374b59a3c46525db7c50db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b532068d2374b59a3c46525db7c50db already in progress: creating tablet
W20250112 20:51:37.289515 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 577dc789f0cb48f49168124b37a6fba0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577dc789f0cb48f49168124b37a6fba0 already in progress: creating tablet
I20250112 20:51:37.295403 1184 ts_tablet_manager.cc:1918] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.295511 1184 log.cc:1198] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/eda44c48bf9746b59ccdc4d39a41ea91
I20250112 20:51:37.295828 1184 ts_tablet_manager.cc:1939] T eda44c48bf9746b59ccdc4d39a41ea91 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.296738 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet eda44c48bf9746b59ccdc4d39a41ea91 (table default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]) successfully deleted
W20250112 20:51:37.297742 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d706f88f14ab4fd9a459944863afaacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d706f88f14ab4fd9a459944863afaacd already in progress: creating tablet
I20250112 20:51:37.298017 1184 tablet_replica.cc:331] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.298199 1184 raft_consensus.cc:2238] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.298354 1184 raft_consensus.cc:2267] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.298728 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1053991511694aa9b914c54e7f01e35a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.299350 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1053991511694aa9b914c54e7f01e35a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.301313 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e3648997eb1241edaf56c825bf76b240
I20250112 20:51:37.301906 446 catalog_manager.cc:6596] ReplaceTablet: tablet e3648997eb1241edaf56c825bf76b240 deleted and replaced by tablet 9a23281d9b6f47a5bf627374fa0cfbf2
I20250112 20:51:37.302132 446 master_service.cc:946] ReplaceTablet: received request to replace tablet b30a34c1abcb4f72a7f4f8a8f5be3720 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.302680 883 tablet_service.cc:1514] Processing DeleteTablet for tablet e3648997eb1241edaf56c825bf76b240 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a23281d9b6f47a5bf627374fa0cfbf2) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.302943 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e3648997eb1241edaf56c825bf76b240 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a23281d9b6f47a5bf627374fa0cfbf2) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:37.303273 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5aca1ff0fd541bfa96bf7f279fa4e2d consensus state has no leader
I20250112 20:51:37.303887 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet e3648997eb1241edaf56c825bf76b240 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a23281d9b6f47a5bf627374fa0cfbf2) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.306921 1163 tablet_bootstrap.cc:492] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.307056 1163 ts_tablet_manager.cc:1397] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.000s
I20250112 20:51:37.307857 1163 raft_consensus.cc:357] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.309314 1163 raft_consensus.cc:383] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.309370 1163 raft_consensus.cc:738] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.309636 1178 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.309556 1163 consensus_queue.cc:260] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.309773 1178 ts_tablet_manager.cc:1397] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250112 20:51:37.309913 1164 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.310041 1164 ts_tablet_manager.cc:1397] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.133s user 0.012s sys 0.037s
I20250112 20:51:37.310076 1163 ts_tablet_manager.cc:1428] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:37.310340 1163 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.310777 1164 raft_consensus.cc:357] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.310961 1164 raft_consensus.cc:383] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.311017 1164 raft_consensus.cc:738] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.311266 1164 consensus_queue.cc:260] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.311720 1164 ts_tablet_manager.cc:1428] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:37.311995 1164 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.212733 1186 ts_tablet_manager.cc:1918] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.312877 1186 log.cc:1198] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b4ad7cf1e0c74265a4d21931279f2318
I20250112 20:51:37.313215 1186 ts_tablet_manager.cc:1939] T b4ad7cf1e0c74265a4d21931279f2318 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.313118 1178 raft_consensus.cc:357] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.313277 1178 raft_consensus.cc:383] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.313313 1178 raft_consensus.cc:738] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.313472 1178 consensus_queue.cc:260] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.313880 1178 ts_tablet_manager.cc:1428] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:37.316234 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b4ad7cf1e0c74265a4d21931279f2318 (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
I20250112 20:51:37.316380 1178 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.320008 1178 tablet_bootstrap.cc:654] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.320879 1163 tablet_bootstrap.cc:654] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.307425 1184 ts_tablet_manager.cc:1905] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.330192 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:37.330482 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]). Expected version 2 got 0
I20250112 20:51:37.330456 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8098069639a2435ab500908db9ecd12a (DEFAULT_TABLE table=default.Table_9a9d57f9339042c0bdc18fd15adae7a3 [id=554666ac0524458dbb36e43d52dfbdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.330808 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8098069639a2435ab500908db9ecd12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.332854 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170] alter complete (version 1)
I20250112 20:51:37.332957 1186 tablet_replica.cc:331] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.333190 1186 raft_consensus.cc:2238] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.333243 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_604c0839efdd41cc8400741910f59256 [id=d46a20ad8b99492395771d94f5dc60fb] alter complete (version 1)
W20250112 20:51:37.334147 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b532068d2374b59a3c46525db7c50db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b532068d2374b59a3c46525db7c50db already in progress: creating tablet
W20250112 20:51:37.334435 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 577dc789f0cb48f49168124b37a6fba0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577dc789f0cb48f49168124b37a6fba0 already in progress: creating tablet
I20250112 20:51:37.338943 874 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 (DEFAULT_TABLE table=default.Table_7265385e3f78422c91c8622a9305378b [id=b0f85fde74d743e798b981f36e27c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.339190 1185 ts_tablet_manager.cc:1918] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.339321 1185 log.cc:1198] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f0daaeb0bf6046da8de969b1f4c71c84
I20250112 20:51:37.339331 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcaaa7af46c48fbb69589e6013968c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.339648 1185 ts_tablet_manager.cc:1939] T f0daaeb0bf6046da8de969b1f4c71c84 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.339730 878 tablet_service.cc:1467] Processing CreateTablet for tablet e92d58058511437d830a8f3cbba1e206 (DEFAULT_TABLE table=default.Table_6b2acbe71aac4af0978c56cb845b5dee [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.340045 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92d58058511437d830a8f3cbba1e206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.340675 1185 tablet_replica.cc:331] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.340868 1185 raft_consensus.cc:2238] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.341053 1185 pending_rounds.cc:62] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:37.341089 1185 pending_rounds.cc:69] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585036712599552 op_type: NO_OP noop_request { }
I20250112 20:51:37.341173 1185 raft_consensus.cc:2883] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.341236 1185 raft_consensus.cc:2267] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.342132 1164 tablet_bootstrap.cc:654] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.343544 877 tablet_service.cc:1467] Processing CreateTablet for tablet 2aef567a7cf64178b53625ac4638048b (DEFAULT_TABLE table=default.Table_d45ae394dc7849ca9b6f6a8bb647dce0 [id=199472dde7de4f88b628efdb7b8120af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.343660 1185 ts_tablet_manager.cc:1905] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.343827 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aef567a7cf64178b53625ac4638048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.346052 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e92d58058511437d830a8f3cbba1e206 (DEFAULT_TABLE table=default.Table_6b2acbe71aac4af0978c56cb845b5dee [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.346164 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 (DEFAULT_TABLE table=default.Table_7265385e3f78422c91c8622a9305378b [id=b0f85fde74d743e798b981f36e27c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.346414 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92d58058511437d830a8f3cbba1e206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.346702 1022 tablet_service.cc:1467] Processing CreateTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 (DEFAULT_TABLE table=default.Table_230aea5bb3074c90aa9e772ab27c1bce [id=5531b293c7c54f58a90422bdd0e6ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.346956 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa091fe2d3c4f72815f228ac3e39128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.348878 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f636df32bd34438bb54989736eaa431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.348918 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8250621cbe45eba669b5ef14e672c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.349049 893 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0578685a045c0801e875f8f42cb0f (DEFAULT_TABLE table=default.Table_2734d2857154455f9cbdcd8e5f06b1f2 [id=513cc1e6ac3940c9ac4d5adf09eedb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.349193 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcaaa7af46c48fbb69589e6013968c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.349361 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b0578685a045c0801e875f8f42cb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.353017 888 tablet_service.cc:1467] Processing CreateTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 (DEFAULT_TABLE table=default.Table_230aea5bb3074c90aa9e772ab27c1bce [id=5531b293c7c54f58a90422bdd0e6ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.353303 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa091fe2d3c4f72815f228ac3e39128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.355634 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8098069639a2435ab500908db9ecd12a (DEFAULT_TABLE table=default.Table_9a9d57f9339042c0bdc18fd15adae7a3 [id=554666ac0524458dbb36e43d52dfbdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.355983 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8098069639a2435ab500908db9ecd12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.357024 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2aef567a7cf64178b53625ac4638048b (DEFAULT_TABLE table=default.Table_d45ae394dc7849ca9b6f6a8bb647dce0 [id=199472dde7de4f88b628efdb7b8120af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.357023 879 tablet_service.cc:1467] Processing CreateTablet for tablet c45ac747ab7f4990a03a39553424e951 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.357296 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aef567a7cf64178b53625ac4638048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.357311 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ac747ab7f4990a03a39553424e951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.360318 1185 ts_tablet_manager.cc:1918] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.360407 1185 log.cc:1198] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/eda44c48bf9746b59ccdc4d39a41ea91
I20250112 20:51:37.360448 1012 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0578685a045c0801e875f8f42cb0f (DEFAULT_TABLE table=default.Table_2734d2857154455f9cbdcd8e5f06b1f2 [id=513cc1e6ac3940c9ac4d5adf09eedb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.360628 737 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0578685a045c0801e875f8f42cb0f (DEFAULT_TABLE table=default.Table_2734d2857154455f9cbdcd8e5f06b1f2 [id=513cc1e6ac3940c9ac4d5adf09eedb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.361030 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b0578685a045c0801e875f8f42cb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.363711 1164 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.363843 1164 ts_tablet_manager.cc:1397] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20250112 20:51:37.364585 1164 raft_consensus.cc:357] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.364799 1164 raft_consensus.cc:383] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.364890 1164 raft_consensus.cc:738] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.365041 739 tablet_service.cc:1467] Processing CreateTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 (DEFAULT_TABLE table=default.Table_230aea5bb3074c90aa9e772ab27c1bce [id=5531b293c7c54f58a90422bdd0e6ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.365129 1164 consensus_queue.cc:260] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.365345 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa091fe2d3c4f72815f228ac3e39128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.365643 1164 ts_tablet_manager.cc:1428] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.365897 1164 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.367960 1164 tablet_bootstrap.cc:654] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.360751 1185 ts_tablet_manager.cc:1939] T eda44c48bf9746b59ccdc4d39a41ea91 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.369027 747 tablet_service.cc:1467] Processing CreateTablet for tablet e92d58058511437d830a8f3cbba1e206 (DEFAULT_TABLE table=default.Table_6b2acbe71aac4af0978c56cb845b5dee [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.369300 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92d58058511437d830a8f3cbba1e206. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.369815 1185 tablet_replica.cc:331] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.369987 1185 raft_consensus.cc:2238] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.370101 1185 raft_consensus.cc:2267] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.370571 1185 ts_tablet_manager.cc:1905] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.370736 1178 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.370854 1178 ts_tablet_manager.cc:1397] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.006s
I20250112 20:51:37.371531 1178 raft_consensus.cc:357] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.371693 1178 raft_consensus.cc:383] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.371735 1178 raft_consensus.cc:738] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.371940 1178 consensus_queue.cc:260] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.372368 1178 ts_tablet_manager.cc:1428] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:37.372608 1178 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.373380 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 (DEFAULT_TABLE table=default.Table_7265385e3f78422c91c8622a9305378b [id=b0f85fde74d743e798b981f36e27c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.373729 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fcaaa7af46c48fbb69589e6013968c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.375993 1178 tablet_bootstrap.cc:654] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.378369 1164 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.378492 1164 ts_tablet_manager.cc:1397] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250112 20:51:37.379173 1164 raft_consensus.cc:357] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.379359 1164 raft_consensus.cc:383] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.379424 1164 raft_consensus.cc:738] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.379606 1164 consensus_queue.cc:260] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.380061 1164 ts_tablet_manager.cc:1428] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:37.380296 1164 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.382196 1164 tablet_bootstrap.cc:654] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.383162 1185 ts_tablet_manager.cc:1918] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.383316 1185 log.cc:1198] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/2b8250621cbe45eba669b5ef14e672c2
I20250112 20:51:37.383647 1185 ts_tablet_manager.cc:1939] T 2b8250621cbe45eba669b5ef14e672c2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.384552 1185 tablet_replica.cc:331] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.387171 1185 raft_consensus.cc:2238] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.387470 1185 pending_rounds.cc:62] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:37.387766 1185 pending_rounds.cc:69] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585034730762240 op_type: NO_OP noop_request { }
I20250112 20:51:37.388046 1185 raft_consensus.cc:2883] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.388283 1185 raft_consensus.cc:2267] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.389807 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b0578685a045c0801e875f8f42cb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.390228 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 2aef567a7cf64178b53625ac4638048b (DEFAULT_TABLE table=default.Table_d45ae394dc7849ca9b6f6a8bb647dce0 [id=199472dde7de4f88b628efdb7b8120af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.390573 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aef567a7cf64178b53625ac4638048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.391180 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 8098069639a2435ab500908db9ecd12a (DEFAULT_TABLE table=default.Table_9a9d57f9339042c0bdc18fd15adae7a3 [id=554666ac0524458dbb36e43d52dfbdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.392761 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8098069639a2435ab500908db9ecd12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.393487 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f0daaeb0bf6046da8de969b1f4c71c84 (table default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]) successfully deleted
W20250112 20:51:37.393801 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 691e2c3f72ab4f439678d166b9b31661 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691e2c3f72ab4f439678d166b9b31661 already in progress: creating tablet
I20250112 20:51:37.394004 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet eda44c48bf9746b59ccdc4d39a41ea91 (table default.Table_558ea8a4fda94f119e0603dcb4ceff52 [id=d1aa966214b04517b08aa1d7937f65aa]) successfully deleted
W20250112 20:51:37.394234 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 104d6b4effb84314a6b2b9db883b4366 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104d6b4effb84314a6b2b9db883b4366 already in progress: creating tablet
I20250112 20:51:37.394416 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 2b8250621cbe45eba669b5ef14e672c2 (table default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]) successfully deleted
I20250112 20:51:37.394433 1185 ts_tablet_manager.cc:1905] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.396731 1178 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.396901 1178 ts_tablet_manager.cc:1397] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250112 20:51:37.397502 1178 raft_consensus.cc:357] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.397666 1178 raft_consensus.cc:383] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.397720 1178 raft_consensus.cc:738] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.397843 877 tablet_service.cc:1467] Processing CreateTablet for tablet 60408b7819484605a95cede93f38a9d5 (DEFAULT_TABLE table=default.Table_47bcd1dceb5441d4a1a05819fb432d3a [id=8626d7cfbdac4e938e8fee46342fbd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.398109 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.398169 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60408b7819484605a95cede93f38a9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.398253 749 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.398322 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.398449 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.398687 754 tablet_service.cc:1467] Processing CreateTablet for tablet c45ac747ab7f4990a03a39553424e951 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.398809 745 tablet_service.cc:1467] Processing CreateTablet for tablet 60408b7819484605a95cede93f38a9d5 (DEFAULT_TABLE table=default.Table_47bcd1dceb5441d4a1a05819fb432d3a [id=8626d7cfbdac4e938e8fee46342fbd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.399014 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ac747ab7f4990a03a39553424e951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.399003 888 tablet_service.cc:1467] Processing CreateTablet for tablet cfebc64ba7dd4631a4375106febb7010 (DEFAULT_TABLE table=default.Table_dae7a8c6178b4a5aab35a17c2abd509f [id=e6657977fdbe4d2cac08d45af1ba0928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.399283 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfebc64ba7dd4631a4375106febb7010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.399256 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf2d3a337ce40a99daafa94e159ab8c (DEFAULT_TABLE table=default.Table_8d0fe81bfac94ece91fa136594e0ae88 [id=27a796b6b826415c8d66cd1116e096a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.399559 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2d3a337ce40a99daafa94e159ab8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.399945 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf2d3a337ce40a99daafa94e159ab8c (DEFAULT_TABLE table=default.Table_8d0fe81bfac94ece91fa136594e0ae88 [id=27a796b6b826415c8d66cd1116e096a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.400210 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2d3a337ce40a99daafa94e159ab8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.400878 1186 pending_rounds.cc:62] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:37.400925 1186 pending_rounds.cc:69] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585036492963840 op_type: NO_OP noop_request { }
I20250112 20:51:37.401019 1186 raft_consensus.cc:2883] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.401100 1186 raft_consensus.cc:2267] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.401746 1178 consensus_queue.cc:260] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.402252 1178 ts_tablet_manager.cc:1428] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 20:51:37.402436 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b30a34c1abcb4f72a7f4f8a8f5be3720
I20250112 20:51:37.402474 1178 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.402884 446 catalog_manager.cc:6596] ReplaceTablet: tablet b30a34c1abcb4f72a7f4f8a8f5be3720 deleted and replaced by tablet 782a195e68124cd391d0394bcb5ec8b2
I20250112 20:51:37.406378 1982 raft_consensus.cc:491] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.406491 1982 raft_consensus.cc:513] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.406627 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet dc3a9740415140fe8864d54f477af954
I20250112 20:51:37.406862 1186 ts_tablet_manager.cc:1905] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.407058 1982 leader_election.cc:290] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:37.407513 749 tablet_service.cc:1467] Processing CreateTablet for tablet cfebc64ba7dd4631a4375106febb7010 (DEFAULT_TABLE table=default.Table_dae7a8c6178b4a5aab35a17c2abd509f [id=e6657977fdbe4d2cac08d45af1ba0928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.407677 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60408b7819484605a95cede93f38a9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.413269 1178 tablet_bootstrap.cc:654] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.415552 1186 ts_tablet_manager.cc:1918] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.415668 1186 log.cc:1198] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/572172a274494e429b7e982815897e61
I20250112 20:51:37.416026 1186 ts_tablet_manager.cc:1939] T 572172a274494e429b7e982815897e61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.417698 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfebc64ba7dd4631a4375106febb7010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.410315 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:37.419252 775 raft_consensus.cc:2463] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:37.419296 1186 tablet_replica.cc:331] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.419468 1186 raft_consensus.cc:2238] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.419595 1186 raft_consensus.cc:2267] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.420069 1186 ts_tablet_manager.cc:1905] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.420305 847 leader_election.cc:304] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.420814 2108 raft_consensus.cc:2798] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:37.420900 2108 raft_consensus.cc:491] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:37.420974 2108 raft_consensus.cc:3054] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.424206 2108 raft_consensus.cc:513] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.437286 1186 ts_tablet_manager.cc:1918] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.437793 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c45ac747ab7f4990a03a39553424e951 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.438077 1186 log.cc:1198] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f0daaeb0bf6046da8de969b1f4c71c84
I20250112 20:51:37.438488 1186 ts_tablet_manager.cc:1939] T f0daaeb0bf6046da8de969b1f4c71c84 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.439447 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.440241 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ac747ab7f4990a03a39553424e951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.440274 1164 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.440982 1025 tablet_service.cc:1467] Processing CreateTablet for tablet cfebc64ba7dd4631a4375106febb7010 (DEFAULT_TABLE table=default.Table_dae7a8c6178b4a5aab35a17c2abd509f [id=e6657977fdbe4d2cac08d45af1ba0928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.441154 2108 leader_election.cc:290] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:37.441352 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfebc64ba7dd4631a4375106febb7010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.442272 1164 ts_tablet_manager.cc:1397] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.017s
W20250112 20:51:37.443219 847 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.443663 1186 tablet_replica.cc:331] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.443877 1186 raft_consensus.cc:2238] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.444064 1186 raft_consensus.cc:2267] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.444664 1186 ts_tablet_manager.cc:1905] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.442917 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:37.445559 775 raft_consensus.cc:3054] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.447912 455 catalog_manager.cc:5526] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:37.448611 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:37.448998 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:37.451022 2059 raft_consensus.cc:491] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.451160 2059 raft_consensus.cc:513] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.451758 2059 leader_election.cc:290] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.452404 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d8e68746cf444ebac5e22728ab6f34" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:37.452929 710 leader_election.cc:343] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.453707 1186 ts_tablet_manager.cc:1918] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:37.453867 1186 log.cc:1198] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/33a90f801d854d4d8f7f09112e2d2929
I20250112 20:51:37.454291 1186 ts_tablet_manager.cc:1939] T 33a90f801d854d4d8f7f09112e2d2929 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.455225 1178 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.455358 1178 ts_tablet_manager.cc:1397] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.010s
I20250112 20:51:37.456224 1178 raft_consensus.cc:357] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.456408 1178 raft_consensus.cc:383] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.456470 1178 raft_consensus.cc:738] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.456666 1178 consensus_queue.cc:260] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.457165 1178 ts_tablet_manager.cc:1428] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.457418 1178 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.458206 1185 ts_tablet_manager.cc:1918] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.458310 1185 log.cc:1198] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/6fe28f23448c47d3979f1982f939e39c
I20250112 20:51:37.458640 1185 ts_tablet_manager.cc:1939] T 6fe28f23448c47d3979f1982f939e39c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.459230 775 raft_consensus.cc:2463] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:37.459595 847 leader_election.cc:304] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.460549 1982 raft_consensus.cc:2798] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.460659 1982 raft_consensus.cc:695] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:37.460899 1982 consensus_queue.cc:237] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.463158 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf2d3a337ce40a99daafa94e159ab8c (DEFAULT_TABLE table=default.Table_8d0fe81bfac94ece91fa136594e0ae88 [id=27a796b6b826415c8d66cd1116e096a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.463681 1178 tablet_bootstrap.cc:654] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.463748 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 60408b7819484605a95cede93f38a9d5 (DEFAULT_TABLE table=default.Table_47bcd1dceb5441d4a1a05819fb432d3a [id=8626d7cfbdac4e938e8fee46342fbd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.464116 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60408b7819484605a95cede93f38a9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.466409 1185 tablet_replica.cc:331] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.466622 1185 raft_consensus.cc:2238] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.466905 1185 raft_consensus.cc:2267] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.467381 1185 ts_tablet_manager.cc:1905] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.469197 893 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.469952 888 tablet_service.cc:1467] Processing CreateTablet for tablet 7a185654a21340bfb877caebb7b13063 (DEFAULT_TABLE table=default.Table_1053991511694aa9b914c54e7f01e35a [id=a25b15c658414878b7509b38712009d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.470295 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a185654a21340bfb877caebb7b13063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.470539 886 tablet_service.cc:1467] Processing CreateTablet for tablet 2363610c772b4a92bce5d4428f5f5c8e (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.470844 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363610c772b4a92bce5d4428f5f5c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.472838 745 tablet_service.cc:1467] Processing CreateTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.473001 874 tablet_service.cc:1467] Processing CreateTablet for tablet 5e137112722545bbae406cf54347aa8c (DEFAULT_TABLE table=default.Table_f4d062a92b0d473e8d60d7224cbc4f9c [id=58d4dcad1a324502b5c485e7a4c5cbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.473047 754 tablet_service.cc:1467] Processing CreateTablet for tablet e5ae913686e349849b407e9ec9d6ef0b (DEFAULT_TABLE table=default.Table_c186d9fb7acb41a8950a79e3d23262a1 [id=64f13654c9b34ddb974bfee4e88c4a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.473194 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762388f687dd4b05ae6ffdaf8c64c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.473299 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e137112722545bbae406cf54347aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.473337 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ae913686e349849b407e9ec9d6ef0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.473477 739 tablet_service.cc:1467] Processing CreateTablet for tablet d44c14aead8249a291b194f228c5ca2b (DEFAULT_TABLE table=default.Table_91d16ee866374194947b9edddcf2e992 [id=d0c3671b61b949e9b4446a46fefaf151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.473752 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44c14aead8249a291b194f228c5ca2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.476997 879 tablet_service.cc:1467] Processing CreateTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.476996 738 tablet_service.cc:1467] Processing CreateTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 (DEFAULT_TABLE table=default.Table_e07c52b8ba4040188dcdd21d2ffa00a6 [id=923465b8c7764a309e44d9126a9864ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.477317 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762388f687dd4b05ae6ffdaf8c64c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.477324 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ade1d7332e45dbaffe3d2d2382f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.479051 887 tablet_service.cc:1467] Processing CreateTablet for tablet 05dc96770e484cfc8ca8388114ddbcf4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.479336 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05dc96770e484cfc8ca8388114ddbcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.480019 884 tablet_service.cc:1467] Processing CreateTablet for tablet 60bbea0f31674fca89b9e98ef12674e8 (DEFAULT_TABLE table=default.Table_93052f6902b2404d9a83652fca013746 [id=54fd7c9cf7a944d79ce455c2ab86fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.480296 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60bbea0f31674fca89b9e98ef12674e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.481014 889 tablet_service.cc:1467] Processing CreateTablet for tablet e5ae913686e349849b407e9ec9d6ef0b (DEFAULT_TABLE table=default.Table_c186d9fb7acb41a8950a79e3d23262a1 [id=64f13654c9b34ddb974bfee4e88c4a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.481269 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ae913686e349849b407e9ec9d6ef0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.483282 911 raft_consensus.cc:3054] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.484994 880 tablet_service.cc:1467] Processing CreateTablet for tablet be1f12369c894682b7d30de4e37bd466 (DEFAULT_TABLE table=default.Table_5c98d71957e443d48bf632f3fe662c5c [id=3a57e9e997904845bd32bfdc2ae17d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.485280 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f12369c894682b7d30de4e37bd466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.486536 2059 consensus_queue.cc:1035] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:37.487551 2076 raft_consensus.cc:491] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.487653 2076 raft_consensus.cc:513] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.488026 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7a185654a21340bfb877caebb7b13063 (DEFAULT_TABLE table=default.Table_1053991511694aa9b914c54e7f01e35a [id=a25b15c658414878b7509b38712009d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.488090 746 tablet_service.cc:1467] Processing CreateTablet for tablet be1f12369c894682b7d30de4e37bd466 (DEFAULT_TABLE table=default.Table_5c98d71957e443d48bf632f3fe662c5c [id=3a57e9e997904845bd32bfdc2ae17d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.488287 2076 leader_election.cc:290] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:37.488294 749 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.488355 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f12369c894682b7d30de4e37bd466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.489005 890 tablet_service.cc:1467] Processing CreateTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 (DEFAULT_TABLE table=default.Table_e07c52b8ba4040188dcdd21d2ffa00a6 [id=923465b8c7764a309e44d9126a9864ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.489305 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ade1d7332e45dbaffe3d2d2382f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.489558 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2d3a337ce40a99daafa94e159ab8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.491442 753 tablet_service.cc:1467] Processing CreateTablet for tablet a89187312df241109454241edcfd1335 (DEFAULT_TABLE table=default.Table_d030f9e0a0474434bb26dd47e6a66afe [id=0f7ac7c4eb144278b96b2fa7e8c0df5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.492015 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89187312df241109454241edcfd1335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.492722 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a185654a21340bfb877caebb7b13063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.491681 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ca4d49496e42ddb263264ceef2f27a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:37.492887 775 raft_consensus.cc:2463] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:37.492990 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.493288 847 leader_election.cc:304] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.493330 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0092258e094a38ba7e223be03d4c9a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:37.493502 882 tablet_service.cc:1467] Processing CreateTablet for tablet 1137357ea95e4be780fa8eaddee092f4 (DEFAULT_TABLE table=default.Table_2a7113eb6fe54951860e36d9f3385a5a [id=f6b92177739d41deb01a6674d763a975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.493815 2076 raft_consensus.cc:2798] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:37.493883 2076 raft_consensus.cc:491] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:37.493942 2076 raft_consensus.cc:3054] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.494032 893 tablet_service.cc:1467] Processing CreateTablet for tablet a89187312df241109454241edcfd1335 (DEFAULT_TABLE table=default.Table_d030f9e0a0474434bb26dd47e6a66afe [id=0f7ac7c4eb144278b96b2fa7e8c0df5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.494587 877 tablet_service.cc:1467] Processing CreateTablet for tablet d44c14aead8249a291b194f228c5ca2b (DEFAULT_TABLE table=default.Table_91d16ee866374194947b9edddcf2e992 [id=d0c3671b61b949e9b4446a46fefaf151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.493307 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a23281d9b6f47a5bf627374fa0cfbf2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:37.495224 847 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.496273 2076 raft_consensus.cc:513] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.497295 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ca4d49496e42ddb263264ceef2f27a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.497382 2076 leader_election.cc:290] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:37.498553 847 leader_election.cc:343] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.498879 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1137357ea95e4be780fa8eaddee092f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.499660 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ca4d49496e42ddb263264ceef2f27a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:37.499755 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89187312df241109454241edcfd1335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.499832 775 raft_consensus.cc:3054] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.500741 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ca4d49496e42ddb263264ceef2f27a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:37.502457 775 raft_consensus.cc:2463] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:37.502943 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.505803 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a89187312df241109454241edcfd1335 (DEFAULT_TABLE table=default.Table_d030f9e0a0474434bb26dd47e6a66afe [id=0f7ac7c4eb144278b96b2fa7e8c0df5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.505930 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.506078 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.506294 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d44c14aead8249a291b194f228c5ca2b (DEFAULT_TABLE table=default.Table_91d16ee866374194947b9edddcf2e992 [id=d0c3671b61b949e9b4446a46fefaf151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.506636 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44c14aead8249a291b194f228c5ca2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.507123 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet f4cae067df1a40098f81cfbd17224c75 with delete_type TABLET_DATA_DELETED (replaced by tablet 4d29316b9d2441bca97f56bdb0e56f0d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.507256 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5e137112722545bbae406cf54347aa8c (DEFAULT_TABLE table=default.Table_f4d062a92b0d473e8d60d7224cbc4f9c [id=58d4dcad1a324502b5c485e7a4c5cbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.507463 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89187312df241109454241edcfd1335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.507773 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e5ae913686e349849b407e9ec9d6ef0b (DEFAULT_TABLE table=default.Table_c186d9fb7acb41a8950a79e3d23262a1 [id=64f13654c9b34ddb974bfee4e88c4a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.509398 737 tablet_service.cc:1467] Processing CreateTablet for tablet 5e137112722545bbae406cf54347aa8c (DEFAULT_TABLE table=default.Table_f4d062a92b0d473e8d60d7224cbc4f9c [id=58d4dcad1a324502b5c485e7a4c5cbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.509466 847 leader_election.cc:304] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.509753 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e137112722545bbae406cf54347aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:37.509851 847 leader_election.cc:343] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.512033 2076 raft_consensus.cc:2798] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.513723 749 tablet_service.cc:1467] Processing CreateTablet for tablet 60bbea0f31674fca89b9e98ef12674e8 (DEFAULT_TABLE table=default.Table_93052f6902b2404d9a83652fca013746 [id=54fd7c9cf7a944d79ce455c2ab86fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.514057 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60bbea0f31674fca89b9e98ef12674e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.515813 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 7a185654a21340bfb877caebb7b13063 (DEFAULT_TABLE table=default.Table_1053991511694aa9b914c54e7f01e35a [id=a25b15c658414878b7509b38712009d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.516175 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e137112722545bbae406cf54347aa8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.516368 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ae913686e349849b407e9ec9d6ef0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.516839 878 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.518517 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1137357ea95e4be780fa8eaddee092f4 (DEFAULT_TABLE table=default.Table_2a7113eb6fe54951860e36d9f3385a5a [id=f6b92177739d41deb01a6674d763a975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.518855 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1137357ea95e4be780fa8eaddee092f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.519835 738 tablet_service.cc:1467] Processing CreateTablet for tablet 05dc96770e484cfc8ca8388114ddbcf4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.520157 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05dc96770e484cfc8ca8388114ddbcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.520658 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d44c14aead8249a291b194f228c5ca2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.521041 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2363610c772b4a92bce5d4428f5f5c8e (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.521397 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363610c772b4a92bce5d4428f5f5c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.521713 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.521982 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 (DEFAULT_TABLE table=default.Table_e07c52b8ba4040188dcdd21d2ffa00a6 [id=923465b8c7764a309e44d9126a9864ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.522389 1027 tablet_service.cc:1467] Processing CreateTablet for tablet be1f12369c894682b7d30de4e37bd466 (DEFAULT_TABLE table=default.Table_5c98d71957e443d48bf632f3fe662c5c [id=3a57e9e997904845bd32bfdc2ae17d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.525012 1163 tablet_bootstrap.cc:492] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.525141 1163 ts_tablet_manager.cc:1397] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.215s user 0.001s sys 0.005s
I20250112 20:51:37.525831 1163 raft_consensus.cc:357] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.526016 1163 raft_consensus.cc:383] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.526065 1163 raft_consensus.cc:738] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.528502 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a185654a21340bfb877caebb7b13063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.529050 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ade1d7332e45dbaffe3d2d2382f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.529623 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762388f687dd4b05ae6ffdaf8c64c999. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.530367 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f12369c894682b7d30de4e37bd466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.533264 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 1137357ea95e4be780fa8eaddee092f4 (DEFAULT_TABLE table=default.Table_2a7113eb6fe54951860e36d9f3385a5a [id=f6b92177739d41deb01a6674d763a975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.534859 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1137357ea95e4be780fa8eaddee092f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.537825 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 60bbea0f31674fca89b9e98ef12674e8 (DEFAULT_TABLE table=default.Table_93052f6902b2404d9a83652fca013746 [id=54fd7c9cf7a944d79ce455c2ab86fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.538190 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60bbea0f31674fca89b9e98ef12674e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.539985 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 2363610c772b4a92bce5d4428f5f5c8e (DEFAULT_TABLE table=default.Table_43ae987b5bce459c8214a0ea34b9df01 [id=a41262782d3c41bea2fc1decd8ba5a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.541718 1185 ts_tablet_manager.cc:1918] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:37.541839 1185 log.cc:1198] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/33a90f801d854d4d8f7f09112e2d2929
I20250112 20:51:37.542208 1185 ts_tablet_manager.cc:1939] T 33a90f801d854d4d8f7f09112e2d2929 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.549778 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 6fe28f23448c47d3979f1982f939e39c (table default.Table_c4712f27518944e0b895370616210edb [id=7c37aefaf9bc42d1b420c9812825432d]) successfully deleted
W20250112 20:51:37.555791 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 577dc789f0cb48f49168124b37a6fba0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577dc789f0cb48f49168124b37a6fba0 already in progress: creating tablet
W20250112 20:51:37.556473 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b532068d2374b59a3c46525db7c50db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b532068d2374b59a3c46525db7c50db already in progress: creating tablet
W20250112 20:51:37.557690 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d706f88f14ab4fd9a459944863afaacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d706f88f14ab4fd9a459944863afaacd already in progress: creating tablet
I20250112 20:51:37.553010 1163 consensus_queue.cc:260] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.563058 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.564042 1164 raft_consensus.cc:357] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.564278 1164 raft_consensus.cc:383] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.563068 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d8e68746cf444ebac5e22728ab6f34" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.566936 2197 raft_consensus.cc:491] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.567190 2197 raft_consensus.cc:513] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.567644 2076 raft_consensus.cc:695] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:37.567935 2076 consensus_queue.cc:237] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.568043 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_0a377dabb6b14129ba1e8519d731bafb [id=11743480162f4442b00dc54a796e3461]) successfully deleted
W20250112 20:51:37.568264 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1ca8fc4a45fe448b919ac001a7980deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca8fc4a45fe448b919ac001a7980deb already in progress: creating tablet
I20250112 20:51:37.568305 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 05dc96770e484cfc8ca8388114ddbcf4 (DEFAULT_TABLE table=default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.569082 2197 leader_election.cc:290] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:37.569356 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 572172a274494e429b7e982815897e61 (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
W20250112 20:51:37.570158 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 60034.8us
I20250112 20:51:37.571903 1163 ts_tablet_manager.cc:1428] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250112 20:51:37.572600 1163 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.573105 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2363610c772b4a92bce5d4428f5f5c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.566696 1931 consensus_queue.cc:1035] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:37.567104 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 (DEFAULT_TABLE table=default.Table_830c03785e1a464fb83fcb19bcd333e3 [id=c0792c79a4c64f5d9415a4f09db63f48]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:37.575078 710 leader_election.cc:343] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.576674 710 leader_election.cc:304] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:37.577337 1931 raft_consensus.cc:2743] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.577507 1931 raft_consensus.cc:491] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.577615 1931 raft_consensus.cc:513] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.578269 1931 leader_election.cc:290] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.579479 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:37.579905 911 raft_consensus.cc:2463] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:37.580526 710 leader_election.cc:304] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.580890 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a23281d9b6f47a5bf627374fa0cfbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.580885 2059 raft_consensus.cc:2798] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:37.580983 2059 raft_consensus.cc:491] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:37.581053 2059 raft_consensus.cc:3054] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.583815 2059 raft_consensus.cc:513] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.585232 2059 leader_election.cc:290] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.586064 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:37.586259 910 raft_consensus.cc:3054] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:37.569569 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d706f88f14ab4fd9a459944863afaacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d706f88f14ab4fd9a459944863afaacd already in progress: creating tablet
I20250112 20:51:37.587023 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f0daaeb0bf6046da8de969b1f4c71c84 (table default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]) successfully deleted
I20250112 20:51:37.587273 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_0a377dabb6b14129ba1e8519d731bafb [id=11743480162f4442b00dc54a796e3461]) successfully deleted
W20250112 20:51:37.587467 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1ca8fc4a45fe448b919ac001a7980deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca8fc4a45fe448b919ac001a7980deb already in progress: creating tablet
W20250112 20:51:37.587622 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 68b4b312c558437892388344767f0ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b4b312c558437892388344767f0ed4 already in progress: creating tablet
W20250112 20:51:37.587814 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 already in progress: creating tablet
W20250112 20:51:37.588012 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 95d8e68746cf444ebac5e22728ab6f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d8e68746cf444ebac5e22728ab6f34 already in progress: creating tablet
W20250112 20:51:37.588227 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f875f075e98e45499291b519caf03184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f875f075e98e45499291b519caf03184 already in progress: creating tablet
W20250112 20:51:37.588419 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4e1968191ac14099b6035ac487e3e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1968191ac14099b6035ac487e3e6bb already in progress: creating tablet
W20250112 20:51:37.588593 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e3648997eb1241edaf56c825bf76b240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3648997eb1241edaf56c825bf76b240 already in progress: creating tablet
I20250112 20:51:37.591092 910 raft_consensus.cc:2463] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:37.592916 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0ab4d7d01f2641e2a56dfad04316b13e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.593427 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ab4d7d01f2641e2a56dfad04316b13e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.594728 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cea4e75bb1d34fcdb209c15b1cd6738d" } modify_external_catalogs: true
I20250112 20:51:37.594888 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cea4e75bb1d34fcdb209c15b1cd6738d" } modify_external_catalogs: true
I20250112 20:51:37.596254 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0bec6474ab5a44ad91c57be6219a3f10" } modify_external_catalogs: true
I20250112 20:51:37.596431 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0bec6474ab5a44ad91c57be6219a3f10" } modify_external_catalogs: true
I20250112 20:51:37.597852 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d030f9e0a0474434bb26dd47e6a66afe" } modify_external_catalogs: true
I20250112 20:51:37.597942 710 leader_election.cc:304] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.598055 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d030f9e0a0474434bb26dd47e6a66afe" } modify_external_catalogs: true
I20250112 20:51:37.599169 2059 raft_consensus.cc:2798] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.599859 2059 raft_consensus.cc:695] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:37.600286 2059 consensus_queue.cc:237] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.603632 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.604768 1048 raft_consensus.cc:3054] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:37.607664 710 leader_election.cc:343] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.611197 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c186d9fb7acb41a8950a79e3d23262a1" } new_table_name: "default.Table_d3f2faf0c65b4e5a85d4694428f6c9d5" modify_external_catalogs: true
I20250112 20:51:37.611996 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a64bcccf91b84c25b6a7cbae25eedc8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.612560 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a64bcccf91b84c25b6a7cbae25eedc8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.608644 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:37.617278 1163 tablet_bootstrap.cc:654] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.619710 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05dc96770e484cfc8ca8388114ddbcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.620235 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8a11203ca15745c0a89a448e1cfc2cc4
I20250112 20:51:37.621829 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:37.622611 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a89187312df241109454241edcfd1335
W20250112 20:51:37.623936 710 leader_election.cc:343] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.626130 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_47d0e6446c184df28dce16b823202a5b" } new_table_name: "default.Table_e0034007793b4b039c081149df5039c3" modify_external_catalogs: true
I20250112 20:51:37.626636 1164 raft_consensus.cc:738] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.626894 1164 consensus_queue.cc:260] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.627714 1164 ts_tablet_manager.cc:1428] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.185s user 0.002s sys 0.000s
W20250112 20:51:37.628131 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5ae913686e349849b407e9ec9d6ef0b consensus state has no leader
I20250112 20:51:37.631778 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_96bb34aa833040378700d550b97d143e" } new_table_name: "default.Table_e99572bd1321493a8d09c4895804e78e" modify_external_catalogs: true
W20250112 20:51:37.636421 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc90d3722fd14d59b9994655cd30ef2b consensus state has no leader
I20250112 20:51:37.637946 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.639858 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_54cf793124084547a2343f0a71d88796" } modify_external_catalogs: true
I20250112 20:51:37.640045 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_54cf793124084547a2343f0a71d88796" } modify_external_catalogs: true
I20250112 20:51:37.640254 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d75f9b410f4790ac926646a3cf0fd0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:37.641081 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa2f3389061143509354f37ae93ef0c7 consensus state has no leader
I20250112 20:51:37.641539 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2f7947f8e7af4b3799d132d1f226f57a" } new_table_name: "default.Table_86bf39762f4f4650aa99eafae2452344" modify_external_catalogs: true
I20250112 20:51:37.642083 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7f659b016002468f88844b86ed218e32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.642611 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f659b016002468f88844b86ed218e32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.643882 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_56d64a03c356468599c3310df24db01b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:37.643901 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.644711 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d64a03c356468599c3310df24db01b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.645469 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a23281d9b6f47a5bf627374fa0cfbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.645561 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b5c9d97f7893417c9bb1225f856fd737"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.646013 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5c9d97f7893417c9bb1225f856fd737 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.647946 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7293510f40804e2aabccf92986489dab
W20250112 20:51:37.649122 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.650135 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2c470161ee88485e98f09828f7e9f22c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.650597 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c470161ee88485e98f09828f7e9f22c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.651628 1931 raft_consensus.cc:491] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.651741 1931 raft_consensus.cc:513] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.652344 1931 leader_election.cc:290] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.652617 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_917e8a15aa6b4cc28455521407819e42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.653141 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_917e8a15aa6b4cc28455521407819e42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.653155 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:37.653407 911 raft_consensus.cc:2463] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:37.653752 710 leader_election.cc:304] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.654000 1931 raft_consensus.cc:2798] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:37.654074 1931 raft_consensus.cc:491] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:37.654124 1931 raft_consensus.cc:3054] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.656512 1931 raft_consensus.cc:513] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:37.664616 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08cfb20cbf384b559bea4528e5f37bc2 consensus state has no leader
I20250112 20:51:37.668009 1931 leader_election.cc:290] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:37.668388 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:37.668601 910 raft_consensus.cc:3054] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.668973 1931 raft_consensus.cc:491] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.669138 1931 raft_consensus.cc:513] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.671624 1931 leader_election.cc:290] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:37.672773 710 leader_election.cc:343] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: f4cae067df1a40098f81cfbd17224c75
I20250112 20:51:37.675434 910 raft_consensus.cc:2463] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:37.676038 710 leader_election.cc:304] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.676335 1931 raft_consensus.cc:2798] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.676365 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_09bd1e992b194beb825bb3280677c608" } new_table_name: "default.Table_2ce3eaab80fa47539dc4e0fd79c1c80b" modify_external_catalogs: true
I20250112 20:51:37.676493 1931 raft_consensus.cc:695] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:37.676884 1931 consensus_queue.cc:237] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.677086 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2a71d5cf4d36447da49bad300ee1b392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:37.677322 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d1dcab09931f446da585ba49901bb41d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:37.677757 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a71d5cf4d36447da49bad300ee1b392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:37.677845 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1dcab09931f446da585ba49901bb41d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.688041 1184 ts_tablet_manager.cc:1918] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250112 20:51:37.693007 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e584b761d234c5aa0ab875acd8e1e05 consensus state has no leader
I20250112 20:51:37.696051 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.706364 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:37.706699 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d66c55c29de7458bb8ea69b826c567d7 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.706878 1184 log.cc:1198] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/33a90f801d854d4d8f7f09112e2d2929
I20250112 20:51:37.706027 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4cae067df1a40098f81cfbd17224c75" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.708642 1049 raft_consensus.cc:2463] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:37.713768 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_341aa7fe62034251879c0e39f76dd6e0" } modify_external_catalogs: true
I20250112 20:51:37.714025 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_341aa7fe62034251879c0e39f76dd6e0" } modify_external_catalogs: true
I20250112 20:51:37.721074 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:45153: Network error: Client connection negotiation failed: client connection to 127.0.104.60:45153: connect: Connection refused (error 111),
I20250112 20:51:37.721257 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 20:51:37.723886 1184 ts_tablet_manager.cc:1939] T 33a90f801d854d4d8f7f09112e2d2929 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.724339 2213 tablet.cc:1721] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.726402 1185 tablet_replica.cc:331] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.726641 1185 raft_consensus.cc:2238] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.726790 1185 raft_consensus.cc:2267] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.727456 1185 ts_tablet_manager.cc:1905] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.733381 1164 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.734443 1163 tablet_bootstrap.cc:492] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.735059 1164 tablet_bootstrap.cc:654] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.736630 1185 ts_tablet_manager.cc:1918] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.736795 1185 log.cc:1198] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/95d8e68746cf444ebac5e22728ab6f34
I20250112 20:51:37.739116 1185 ts_tablet_manager.cc:1939] T 95d8e68746cf444ebac5e22728ab6f34 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.740407 1185 tablet_replica.cc:331] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.740610 1185 raft_consensus.cc:2238] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.740764 1185 raft_consensus.cc:2267] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.741305 1185 ts_tablet_manager.cc:1905] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.745667 1184 tablet_replica.cc:331] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.745927 1184 raft_consensus.cc:2238] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.746232 1184 pending_rounds.cc:62] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:37.746294 1184 pending_rounds.cc:69] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585037070180352 op_type: NO_OP noop_request { }
I20250112 20:51:37.746410 1184 raft_consensus.cc:2883] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.746519 1184 raft_consensus.cc:2267] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.746953 1163 ts_tablet_manager.cc:1397] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.162s user 0.014s sys 0.000s
I20250112 20:51:37.747901 1163 raft_consensus.cc:357] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.748101 1163 raft_consensus.cc:383] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.748157 1163 raft_consensus.cc:738] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.748360 1163 consensus_queue.cc:260] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.748920 1163 ts_tablet_manager.cc:1428] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.749207 1163 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.750728 1163 tablet_bootstrap.cc:654] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:37.751133 847 leader_election.cc:343] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.751320 847 leader_election.cc:304] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:37.751539 2076 raft_consensus.cc:2743] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.751727 1184 ts_tablet_manager.cc:1905] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.753567 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 (DEFAULT_TABLE table=default.Table_9ff8fdbb27a140239a8286434feff5a2 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.753969 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a195e68124cd391d0394bcb5ec8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.757540 1184 ts_tablet_manager.cc:1918] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.757676 1184 log.cc:1198] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/42eb0f79df5444f8a8360921d5f2b9ee
I20250112 20:51:37.758036 1184 ts_tablet_manager.cc:1939] T 42eb0f79df5444f8a8360921d5f2b9ee P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.759953 1184 tablet_replica.cc:331] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.760195 1184 raft_consensus.cc:2238] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.760344 1184 raft_consensus.cc:2267] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.761839 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]). Expected version 2 got 0
W20250112 20:51:37.768301 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 77656.5us
I20250112 20:51:37.768906 2140 raft_consensus.cc:491] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.769035 2140 raft_consensus.cc:513] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.769701 2140 leader_election.cc:290] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:37.770171 710 leader_election.cc:343] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:37.770452 710 leader_election.cc:343] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.770712 710 leader_election.cc:304] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:37.770875 710 raft_consensus.cc:2681] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:37.773355 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d66c55c29de7458bb8ea69b826c567d7
I20250112 20:51:37.773880 446 catalog_manager.cc:6596] ReplaceTablet: tablet d66c55c29de7458bb8ea69b826c567d7 deleted and replaced by tablet 438e2c14353c4a8cb7242d75bb3ba673
I20250112 20:51:37.774360 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 913149098822406e8d4ca5a440c9afd7
I20250112 20:51:37.780445 754 tablet_service.cc:1467] Processing CreateTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 (DEFAULT_TABLE table=default.Table_9ff8fdbb27a140239a8286434feff5a2 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.780963 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a195e68124cd391d0394bcb5ec8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.783509 1184 ts_tablet_manager.cc:1905] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:37.784420 2204 log.cc:926] Time spent T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250112 20:51:37.792773 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.792667 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.794075 1163 tablet_bootstrap.cc:492] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.794173 1164 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.794217 1163 ts_tablet_manager.cc:1397] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20250112 20:51:37.794396 1164 ts_tablet_manager.cc:1397] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20250112 20:51:37.794978 1163 raft_consensus.cc:357] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.795174 1163 raft_consensus.cc:383] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.795244 1163 raft_consensus.cc:738] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.795282 1164 raft_consensus.cc:357] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.795549 1164 raft_consensus.cc:383] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.795490 1163 consensus_queue.cc:260] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.795637 1164 raft_consensus.cc:738] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.796592 1163 ts_tablet_manager.cc:1428] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.796876 1163 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.796145 1164 consensus_queue.cc:260] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.798355 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.798600 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.798791 1163 tablet_bootstrap.cc:654] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.800495 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572172a274494e429b7e982815897e61" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:37.800690 775 raft_consensus.cc:2388] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:37.803130 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet a89187312df241109454241edcfd1335 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:37.805020 983 leader_election.cc:343] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 572172a274494e429b7e982815897e61
I20250112 20:51:37.805158 983 leader_election.cc:304] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:37.805287 737 tablet_service.cc:1514] Processing DeleteTablet for tablet a89187312df241109454241edcfd1335 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.805348 2140 raft_consensus.cc:3054] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.808540 2140 raft_consensus.cc:2743] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.810496 2223 tablet.cc:1721] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.811110 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293510f40804e2aabccf92986489dab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.812085 1178 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.812211 1178 ts_tablet_manager.cc:1397] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.355s user 0.008s sys 0.014s
I20250112 20:51:37.814126 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.814337 1186 tablet_replica.cc:331] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.814332 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.814496 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.814534 1186 raft_consensus.cc:2238] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.814756 1186 pending_rounds.cc:62] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:37.814821 1186 pending_rounds.cc:69] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585039642288128 op_type: NO_OP noop_request { }
I20250112 20:51:37.814925 1186 raft_consensus.cc:2883] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.815028 1186 raft_consensus.cc:2267] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.815644 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.815837 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.815989 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293510f40804e2aabccf92986489dab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.816846 874 tablet_service.cc:1514] Processing DeleteTablet for tablet a89187312df241109454241edcfd1335 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.816727 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a11203ca15745c0a89a448e1cfc2cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.816344 880 tablet_service.cc:1467] Processing CreateTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 (DEFAULT_TABLE table=default.Table_9ff8fdbb27a140239a8286434feff5a2 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.817298 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 782a195e68124cd391d0394bcb5ec8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.824810 1186 ts_tablet_manager.cc:1905] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.825285 2076 raft_consensus.cc:491] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.825405 2076 raft_consensus.cc:513] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.825929 2076 leader_election.cc:290] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:37.829504 1185 ts_tablet_manager.cc:1918] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.829623 1185 log.cc:1198] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f4cae067df1a40098f81cfbd17224c75
I20250112 20:51:37.829993 1185 ts_tablet_manager.cc:1939] T f4cae067df1a40098f81cfbd17224c75 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.830482 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]). Expected version 1 got 0
I20250112 20:51:37.830900 1185 tablet_replica.cc:331] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.831038 1186 ts_tablet_manager.cc:1918] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.831092 1185 raft_consensus.cc:2238] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.831218 1186 log.cc:1198] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/a3ca4d49496e42ddb263264ceef2f27a
I20250112 20:51:37.831243 1185 raft_consensus.cc:2267] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.831645 1186 ts_tablet_manager.cc:1939] T a3ca4d49496e42ddb263264ceef2f27a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.831789 1185 ts_tablet_manager.cc:1905] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.832238 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 33a90f801d854d4d8f7f09112e2d2929 (table default.Table_0a377dabb6b14129ba1e8519d731bafb [id=11743480162f4442b00dc54a796e3461]) successfully deleted
I20250112 20:51:37.832653 1186 tablet_replica.cc:331] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.832912 1186 raft_consensus.cc:2238] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.832921 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 42eb0f79df5444f8a8360921d5f2b9ee (table default.Table_43f4ea15650c4619afdabdb3c169df86 [id=dd64a2679c0849279e9dae49f268056b]) successfully deleted
I20250112 20:51:37.833137 1186 raft_consensus.cc:2267] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:37.833139 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1ca8fc4a45fe448b919ac001a7980deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca8fc4a45fe448b919ac001a7980deb already in progress: creating tablet
W20250112 20:51:37.833334 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 68b4b312c558437892388344767f0ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b4b312c558437892388344767f0ed4 already in progress: creating tablet
W20250112 20:51:37.833541 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 already in progress: creating tablet
W20250112 20:51:37.833711 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 95d8e68746cf444ebac5e22728ab6f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d8e68746cf444ebac5e22728ab6f34 already in progress: creating tablet
W20250112 20:51:37.833880 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f875f075e98e45499291b519caf03184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f875f075e98e45499291b519caf03184 already in progress: creating tablet
W20250112 20:51:37.834066 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4e1968191ac14099b6035ac487e3e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1968191ac14099b6035ac487e3e6bb already in progress: creating tablet
W20250112 20:51:37.834218 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e3648997eb1241edaf56c825bf76b240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3648997eb1241edaf56c825bf76b240 already in progress: creating tablet
I20250112 20:51:37.834604 1186 ts_tablet_manager.cc:1905] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.834944 2224 tablet.cc:1721] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:37.835062 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 68b4b312c558437892388344767f0ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b4b312c558437892388344767f0ed4 already in progress: creating tablet
W20250112 20:51:37.835250 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 already in progress: creating tablet
I20250112 20:51:37.835423 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 95d8e68746cf444ebac5e22728ab6f34 (table default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]) successfully deleted
W20250112 20:51:37.835602 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f875f075e98e45499291b519caf03184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f875f075e98e45499291b519caf03184 already in progress: creating tablet
W20250112 20:51:37.835789 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4e1968191ac14099b6035ac487e3e6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e1968191ac14099b6035ac487e3e6bb already in progress: creating tablet
W20250112 20:51:37.836014 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e3648997eb1241edaf56c825bf76b240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3648997eb1241edaf56c825bf76b240 already in progress: creating tablet
I20250112 20:51:37.836299 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f4cae067df1a40098f81cfbd17224c75 (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
I20250112 20:51:37.839993 1164 ts_tablet_manager.cc:1428] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.045s user 0.001s sys 0.000s
I20250112 20:51:37.843983 1164 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.836568 1163 tablet_bootstrap.cc:492] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.846398 2225 tablet.cc:1721] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:37.846617 1163 ts_tablet_manager.cc:1397] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20250112 20:51:37.847453 1163 raft_consensus.cc:357] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.847628 1163 raft_consensus.cc:383] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.848012 1163 raft_consensus.cc:738] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.848269 1163 consensus_queue.cc:260] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.848803 1163 ts_tablet_manager.cc:1428] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.852030 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d60604713524dba8f3a42c4f3bc7300" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.852490 1163 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.853868 1163 tablet_bootstrap.cc:654] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.837725 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7293510f40804e2aabccf92986489dab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.810642 1184 ts_tablet_manager.cc:1918] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.854699 1184 log.cc:1198] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8f636df32bd34438bb54989736eaa431
I20250112 20:51:37.855057 1184 ts_tablet_manager.cc:1939] T 8f636df32bd34438bb54989736eaa431 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.812925 1178 raft_consensus.cc:357] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.855376 1178 raft_consensus.cc:383] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.855484 1178 raft_consensus.cc:738] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.855731 1178 consensus_queue.cc:260] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.856285 1178 ts_tablet_manager.cc:1428] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20250112 20:51:37.854252 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d60604713524dba8f3a42c4f3bc7300" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:37.857026 1184 tablet_replica.cc:331] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.857188 1184 raft_consensus.cc:2238] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.857197 775 raft_consensus.cc:2463] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:37.857328 1184 raft_consensus.cc:2267] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.857812 1184 ts_tablet_manager.cc:1905] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.858412 847 leader_election.cc:304] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:37.858695 2076 raft_consensus.cc:2798] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:37.858757 2076 raft_consensus.cc:491] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:37.858810 2076 raft_consensus.cc:3054] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.861063 1178 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.863032 1178 tablet_bootstrap.cc:654] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.863992 2076 raft_consensus.cc:513] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.864552 2076 leader_election.cc:290] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:37.874694 847 leader_election.cc:343] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.875548 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d60604713524dba8f3a42c4f3bc7300" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:37.876292 847 leader_election.cc:343] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.877278 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d60604713524dba8f3a42c4f3bc7300" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:37.877455 774 raft_consensus.cc:3054] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.880195 774 raft_consensus.cc:2463] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:37.881676 1186 ts_tablet_manager.cc:1918] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.881712 1164 tablet_bootstrap.cc:654] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.884399 1184 ts_tablet_manager.cc:1918] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.884656 1186 log.cc:1198] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8d08048c402345a686d1e3fb3039bbc1
W20250112 20:51:37.846361 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet dc3a9740415140fe8864d54f477af954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc3a9740415140fe8864d54f477af954 already in progress: creating tablet
W20250112 20:51:37.885164 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b30a34c1abcb4f72a7f4f8a8f5be3720 already in progress: creating tablet
I20250112 20:51:37.883306 847 leader_election.cc:304] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:37.885553 1186 ts_tablet_manager.cc:1939] T 8d08048c402345a686d1e3fb3039bbc1 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.885565 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet a3ca4d49496e42ddb263264ceef2f27a (table default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]) successfully deleted
I20250112 20:51:37.885749 2076 raft_consensus.cc:2798] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.887081 2076 raft_consensus.cc:695] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:37.887295 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.887387 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8f636df32bd34438bb54989736eaa431 (table default.Table_46fa14b9681540c5a4649563f92f7a53 [id=d096e82f53ee479284d11c23c80e7219]) successfully deleted
I20250112 20:51:37.887569 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 913149098822406e8d4ca5a440c9afd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.887529 2076 consensus_queue.cc:237] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.887795 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 921ae136d5e2438e950b96f156f23da3 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.888319 453 master_service.cc:946] ReplaceTablet: received request to replace tablet d570daec845f40778c48ff3f2e2e30cb from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.889225 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.889393 738 tablet_service.cc:1514] Processing DeleteTablet for tablet d66c55c29de7458bb8ea69b826c567d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 438e2c14353c4a8cb7242d75bb3ba673) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.891077 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f8d521e6c8cd461887f93f395c27b89e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.892452 1186 tablet_replica.cc:331] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.892625 1186 raft_consensus.cc:2238] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.892839 1186 pending_rounds.cc:62] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:37.892884 1186 pending_rounds.cc:69] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585040951119872 op_type: NO_OP noop_request { }
I20250112 20:51:37.892969 1186 raft_consensus.cc:2883] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:37.893038 1186 raft_consensus.cc:2267] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.886862 1163 tablet_bootstrap.cc:492] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.895318 1163 ts_tablet_manager.cc:1397] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250112 20:51:37.896085 1163 raft_consensus.cc:357] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.896250 1163 raft_consensus.cc:383] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.896298 1163 raft_consensus.cc:738] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.896624 1163 consensus_queue.cc:260] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.897677 1163 ts_tablet_manager.cc:1428] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:37.897934 1163 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.899212 1178 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.899336 1178 ts_tablet_manager.cc:1397] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20250112 20:51:37.899375 1163 tablet_bootstrap.cc:654] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.900230 1184 log.cc:1198] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/2b8250621cbe45eba669b5ef14e672c2
I20250112 20:51:37.900233 1178 raft_consensus.cc:357] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.900416 1178 raft_consensus.cc:383] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.900472 1178 raft_consensus.cc:738] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.900862 1178 consensus_queue.cc:260] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.900583 1184 ts_tablet_manager.cc:1939] T 2b8250621cbe45eba669b5ef14e672c2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.902835 1178 ts_tablet_manager.cc:1428] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250112 20:51:37.903076 1178 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:37.903118 452 catalog_manager.cc:5526] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:37.903443 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.903512 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:37.903625 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.903729 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:37.903779 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 913149098822406e8d4ca5a440c9afd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.904289 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d66c55c29de7458bb8ea69b826c567d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 438e2c14353c4a8cb7242d75bb3ba673) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.905683 1178 tablet_bootstrap.cc:654] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.906680 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (table default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2]). Expected version 1 got 0
I20250112 20:51:37.909551 1184 tablet_replica.cc:331] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:37.909729 1184 raft_consensus.cc:2238] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:37.909844 1164 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.909864 1184 raft_consensus.cc:2267] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:37.909983 1164 ts_tablet_manager.cc:1397] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20250112 20:51:37.910374 1184 ts_tablet_manager.cc:1905] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:37.912459 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f8d521e6c8cd461887f93f395c27b89e
I20250112 20:51:37.912906 448 catalog_manager.cc:6596] ReplaceTablet: tablet f8d521e6c8cd461887f93f395c27b89e deleted and replaced by tablet 1a505986bc6146829cbafd0de6682d93
I20250112 20:51:37.914528 738 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 (DEFAULT_TABLE table=default.Table_0ab4d7d01f2641e2a56dfad04316b13e [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.914904 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb9c2e6f37f4184a106aed05bae7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.921306 737 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d521e6c8cd461887f93f395c27b89e with delete_type TABLET_DATA_DELETED (replaced by tablet 1a505986bc6146829cbafd0de6682d93) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.921784 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8d08048c402345a686d1e3fb3039bbc1 (table default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]) successfully deleted
I20250112 20:51:37.921810 1163 tablet_bootstrap.cc:492] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.921952 1163 ts_tablet_manager.cc:1397] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250112 20:51:37.922490 1164 raft_consensus.cc:357] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.922678 1164 raft_consensus.cc:383] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.922730 1164 raft_consensus.cc:738] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.922775 1163 raft_consensus.cc:357] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.922930 1163 raft_consensus.cc:383] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.922999 1163 raft_consensus.cc:738] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.923197 1163 consensus_queue.cc:260] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.923722 1163 ts_tablet_manager.cc:1428] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:37.923974 1163 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.922942 1164 consensus_queue.cc:260] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.925649 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5d48f415092e41a3ada61770f9908c9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:37.926146 1186 ts_tablet_manager.cc:1905] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:37.926409 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d48f415092e41a3ada61770f9908c9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:37.927027 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 (DEFAULT_TABLE table=default.Table_0ab4d7d01f2641e2a56dfad04316b13e [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.927405 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb9c2e6f37f4184a106aed05bae7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.928617 1164 ts_tablet_manager.cc:1428] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250112 20:51:37.928889 1164 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.930415 1164 tablet_bootstrap.cc:654] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.932912 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d521e6c8cd461887f93f395c27b89e with delete_type TABLET_DATA_DELETED (replaced by tablet 1a505986bc6146829cbafd0de6682d93) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:37.935129 2059 raft_consensus.cc:491] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:37.935256 2059 raft_consensus.cc:513] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.935860 2059 leader_election.cc:290] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:37.937053 710 leader_election.cc:343] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 8d08048c402345a686d1e3fb3039bbc1
I20250112 20:51:37.937474 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (table default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2]). Expected version 1 got 0
I20250112 20:51:37.937676 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 85897bfec1834c57956a23c5dfc2e853 (table default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7]). Expected version 1 got 0
I20250112 20:51:37.938271 1163 tablet_bootstrap.cc:654] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.942358 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:37.942672 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet db9bab2477a94c169817d9de8aaad1c0 (table default.Table_c0ad7aebc1e245b2bcd5c4af8c8211c6 [id=2f68c758f6554102961912265cf48921], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet db9bab2477a94c169817d9de8aaad1c0 consensus state has no leader
I20250112 20:51:37.942865 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 2b8250621cbe45eba669b5ef14e672c2 (table default.Table_18e26eeea5ff4c4196d49055411fb3ad [id=4d6ab5fbdfe04c9db5bfd418d9563669]) successfully deleted
I20250112 20:51:37.943099 1184 ts_tablet_manager.cc:1918] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:37.943163 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b30a34c1abcb4f72a7f4f8a8f5be3720 already in progress: creating tablet
W20250112 20:51:37.943385 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet dc3a9740415140fe8864d54f477af954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc3a9740415140fe8864d54f477af954 already in progress: creating tablet
I20250112 20:51:37.943609 1185 ts_tablet_manager.cc:1918] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:37.943725 1185 log.cc:1198] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/a3ca4d49496e42ddb263264ceef2f27a
I20250112 20:51:37.944033 1185 ts_tablet_manager.cc:1939] T a3ca4d49496e42ddb263264ceef2f27a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:37.945302 1178 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:37.945431 1178 ts_tablet_manager.cc:1397] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.003s
I20250112 20:51:37.946702 2235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:45153 every 8 connection(s)
I20250112 20:51:37.947054 1816 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:45153
I20250112 20:51:37.951367 2076 consensus_queue.cc:1035] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:37.953764 1816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-2/data/info.pb
I20250112 20:51:37.957383 1186 ts_tablet_manager.cc:1918] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:37.961913 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 1816
I20250112 20:51:37.967592 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet a3ca4d49496e42ddb263264ceef2f27a (table default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]) successfully deleted
I20250112 20:51:37.968636 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f425779195594599a3ab8c4dc5bee7f9 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:37.972389 1186 log.cc:1198] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1d60604713524dba8f3a42c4f3bc7300
I20250112 20:51:37.973048 1186 ts_tablet_manager.cc:1939] T 1d60604713524dba8f3a42c4f3bc7300 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:37.974387 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:37.979349 1186 tablet_replica.cc:331] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:37.979647 1186 raft_consensus.cc:2238] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:37.979925 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1d60604713524dba8f3a42c4f3bc7300 (table default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]) successfully deleted
I20250112 20:51:37.960884 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d08048c402345a686d1e3fb3039bbc1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:37.961467 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d66c55c29de7458bb8ea69b826c567d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 438e2c14353c4a8cb7242d75bb3ba673) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.963182 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 913149098822406e8d4ca5a440c9afd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.966411 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 (DEFAULT_TABLE table=default.Table_0ab4d7d01f2641e2a56dfad04316b13e [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:37.968206 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 572172a274494e429b7e982815897e61 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d384628ba9b4f81b768eda4f796632b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.968480 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0092258e094a38ba7e223be03d4c9a with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd46a422093495ba8b5b03039874a32) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.970335 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d521e6c8cd461887f93f395c27b89e with delete_type TABLET_DATA_DELETED (replaced by tablet 1a505986bc6146829cbafd0de6682d93) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.970634 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 54de54eabe5946058d1940b79f5fb832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.973440 1163 tablet_bootstrap.cc:492] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:37.980945 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb9c2e6f37f4184a106aed05bae7924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.981017 1163 ts_tablet_manager.cc:1397] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20250112 20:51:37.981858 1163 raft_consensus.cc:357] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.982028 1163 raft_consensus.cc:383] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.982069 1163 raft_consensus.cc:738] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.982268 1163 consensus_queue.cc:260] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.982702 1163 ts_tablet_manager.cc:1428] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:37.982940 1163 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:37.984531 1163 tablet_bootstrap.cc:654] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.985535 1164 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:37.985715 1164 ts_tablet_manager.cc:1397] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250112 20:51:37.986469 1164 raft_consensus.cc:357] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.986737 1164 raft_consensus.cc:383] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.986841 1164 raft_consensus.cc:738] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.987129 1164 consensus_queue.cc:260] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:37.987396 710 leader_election.cc:343] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:37.987504 710 leader_election.cc:304] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:37.987704 2059 raft_consensus.cc:2743] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:37.988199 1164 ts_tablet_manager.cc:1428] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:37.988405 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f425779195594599a3ab8c4dc5bee7f9
I20250112 20:51:37.988618 1164 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:37.988905 452 catalog_manager.cc:6596] ReplaceTablet: tablet f425779195594599a3ab8c4dc5bee7f9 deleted and replaced by tablet 15b009608f514353be8a58408b4abdea
I20250112 20:51:37.990773 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:37.990960 1184 log.cc:1198] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f4cae067df1a40098f81cfbd17224c75
I20250112 20:51:37.991276 1184 ts_tablet_manager.cc:1939] T f4cae067df1a40098f81cfbd17224c75 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:37.991566 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:37.991616 1185 tablet_replica.cc:331] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:37.995862 1048 raft_consensus.cc:3054] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.996347 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f4cae067df1a40098f81cfbd17224c75 (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
W20250112 20:51:37.996616 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8a11203ca15745c0a89a448e1cfc2cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a11203ca15745c0a89a448e1cfc2cc4 already in progress: creating tablet
I20250112 20:51:37.998463 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7038f1cfb0f944378fdf6ab8daa15d60" } modify_external_catalogs: true
I20250112 20:51:37.998709 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7038f1cfb0f944378fdf6ab8daa15d60" } modify_external_catalogs: true
I20250112 20:51:37.998833 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (table default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2]). Expected version 1 got 0
I20250112 20:51:38.001302 1186 pending_rounds.cc:62] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:38.001370 1186 pending_rounds.cc:69] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585039201980416 op_type: NO_OP noop_request { }
I20250112 20:51:37.972214 1178 raft_consensus.cc:357] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:37.990060 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.001686 1178 raft_consensus.cc:383] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.001755 1178 raft_consensus.cc:738] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.002214 1178 consensus_queue.cc:260] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.002718 1178 ts_tablet_manager.cc:1428] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.056s user 0.001s sys 0.000s
I20250112 20:51:38.003007 1178 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.004824 2108 consensus_queue.cc:1035] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.005978 1178 tablet_bootstrap.cc:654] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.006922 1185 raft_consensus.cc:2238] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.007113 1185 raft_consensus.cc:2267] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.007155 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_df6048c1d029498b8d19b7645fa5afd8" } modify_external_catalogs: true
I20250112 20:51:38.007169 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae1eb78e32524283a93e97294e33bf5a" } modify_external_catalogs: true
I20250112 20:51:38.007354 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_df6048c1d029498b8d19b7645fa5afd8" } modify_external_catalogs: true
I20250112 20:51:38.007432 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae1eb78e32524283a93e97294e33bf5a" } modify_external_catalogs: true
I20250112 20:51:38.008983 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0cbb400711764b7a8fffe933e3334d42" } new_table_name: "default.Table_f263ea98be854666aeb1b47042a00e85" modify_external_catalogs: true
I20250112 20:51:38.010856 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_06293bb20aca4d9696355a7fc3fdbe54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.011368 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06293bb20aca4d9696355a7fc3fdbe54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.011591 1185 ts_tablet_manager.cc:1905] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.012552 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_47bcd1dceb5441d4a1a05819fb432d3a" } new_table_name: "default.Table_a58bef1719324f5c87d4e91573f71d90" modify_external_catalogs: true
I20250112 20:51:38.013676 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0ab4d7d01f2641e2a56dfad04316b13e" } new_table_name: "default.Table_eba10319e22144a4bb6eee085519a0fb" modify_external_catalogs: true
I20250112 20:51:38.014277 1164 tablet_bootstrap.cc:654] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.001477 1186 raft_consensus.cc:2883] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:38.015245 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.015393 1186 raft_consensus.cc:2267] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.016234 1186 ts_tablet_manager.cc:1905] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:38.017428 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8c5b10576bd34af5b02f8d8dcc3759ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c5b10576bd34af5b02f8d8dcc3759ab already in progress: creating tablet
I20250112 20:51:38.020694 2108 raft_consensus.cc:491] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.020857 2108 raft_consensus.cc:513] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.021483 2108 leader_election.cc:290] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.021931 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.022121 1046 raft_consensus.cc:2463] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.022073 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.022259 774 raft_consensus.cc:2463] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.023872 1184 tablet_replica.cc:331] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.024086 1184 raft_consensus.cc:2238] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.024298 1184 raft_consensus.cc:2267] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.024986 847 leader_election.cc:304] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.025127 1184 ts_tablet_manager.cc:1905] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.025323 2108 raft_consensus.cc:2798] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.025395 2108 raft_consensus.cc:491] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.025465 2108 raft_consensus.cc:3054] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:38.027184 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a89187312df241109454241edcfd1335 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89187312df241109454241edcfd1335 already in progress: creating tablet
I20250112 20:51:38.027951 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet fe1dd81dcdc94420bb12a34c1167da8a
I20250112 20:51:38.028074 2108 raft_consensus.cc:513] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.028592 2108 leader_election.cc:290] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.028667 880 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:38.028959 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dea98c29bc744009ab9b3eeb53684f29 consensus state has no leader
I20250112 20:51:38.029464 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.029667 1046 raft_consensus.cc:3054] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:38.030071 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60408b7819484605a95cede93f38a9d5 consensus state has no leader
I20250112 20:51:38.031337 1163 tablet_bootstrap.cc:492] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:38.031458 1163 ts_tablet_manager.cc:1397] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250112 20:51:38.032174 1163 raft_consensus.cc:357] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.032284 1046 raft_consensus.cc:2463] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.032995 847 leader_election.cc:304] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:38.033320 2108 raft_consensus.cc:2798] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.033993 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4303e4aec9694c49aca7c1532eaab3ed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.034143 2197 raft_consensus.cc:491] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.034384 774 raft_consensus.cc:3054] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.034276 2197 raft_consensus.cc:513] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.034921 2197 leader_election.cc:290] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.035179 2108 raft_consensus.cc:695] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.035300 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d8e68746cf444ebac5e22728ab6f34" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.035557 2108 consensus_queue.cc:237] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:38.035741 847 leader_election.cc:343] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:38.036363 847 leader_election.cc:343] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 95d8e68746cf444ebac5e22728ab6f34
I20250112 20:51:38.036458 847 leader_election.cc:304] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:38.036644 2197 raft_consensus.cc:2743] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:38.036818 1982 raft_consensus.cc:491] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.036921 1982 raft_consensus.cc:513] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.037416 1982 leader_election.cc:290] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.038225 774 raft_consensus.cc:2463] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.041687 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640dc9187c2243248ef73bac6616c126" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.042237 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640dc9187c2243248ef73bac6616c126" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.042454 774 raft_consensus.cc:2463] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.042726 738 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.042798 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1dd81dcdc94420bb12a34c1167da8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.043164 847 leader_election.cc:304] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.043418 1982 raft_consensus.cc:2798] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.043493 1982 raft_consensus.cc:491] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.043550 1982 raft_consensus.cc:3054] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.045933 1982 raft_consensus.cc:513] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.046516 1982 leader_election.cc:290] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.046924 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640dc9187c2243248ef73bac6616c126" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.047030 1163 raft_consensus.cc:383] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.047088 1163 raft_consensus.cc:738] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.049116 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640dc9187c2243248ef73bac6616c126" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.049322 774 raft_consensus.cc:3054] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.047273 1163 consensus_queue.cc:260] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.051271 1046 raft_consensus.cc:2463] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.051878 1163 ts_tablet_manager.cc:1428] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250112 20:51:38.054271 774 raft_consensus.cc:2463] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.055262 1163 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:38.057161 1163 tablet_bootstrap.cc:654] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.057801 847 leader_election.cc:304] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.058071 1982 raft_consensus.cc:2798] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.058176 1982 raft_consensus.cc:695] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.058398 1982 consensus_queue.cc:237] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.065428 2238 sys_catalog.cc:263] Verifying existing consensus state
I20250112 20:51:38.067049 1185 ts_tablet_manager.cc:1918] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.068017 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_acf981f928ef4a12a31795dec2c5256e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.068295 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9494be2cad4b41559c1d1563a64ce78f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.068028 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_acc683b25e9a4c3f84ce1b9476963382"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.069437 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90f38fee7fd8445a98f6af6d299318bc" } modify_external_catalogs: true
I20250112 20:51:38.068970 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_89dea959eeb74e07bf12c140416c0f75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.070592 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc683b25e9a4c3f84ce1b9476963382 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.070742 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9494be2cad4b41559c1d1563a64ce78f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.070770 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acf981f928ef4a12a31795dec2c5256e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.070947 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89dea959eeb74e07bf12c140416c0f75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.073493 1185 log.cc:1198] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/572172a274494e429b7e982815897e61
I20250112 20:51:38.067907 1164 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.073899 1164 ts_tablet_manager.cc:1397] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.003s
I20250112 20:51:38.076988 1185 ts_tablet_manager.cc:1939] T 572172a274494e429b7e982815897e61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.074723 1164 raft_consensus.cc:357] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.078147 1164 raft_consensus.cc:383] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.078239 1164 raft_consensus.cc:738] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.078437 1164 consensus_queue.cc:260] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.078992 1184 ts_tablet_manager.cc:1918] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:38.079121 1184 log.cc:1198] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7293510f40804e2aabccf92986489dab
I20250112 20:51:38.079478 1184 ts_tablet_manager.cc:1939] T 7293510f40804e2aabccf92986489dab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.083356 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.085350 958 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.087087 1164 ts_tablet_manager.cc:1428] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 20:51:38.088181 1164 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.087908 1185 tablet_replica.cc:331] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.088989 1185 raft_consensus.cc:2238] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.092046 1185 raft_consensus.cc:2267] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.092661 1185 ts_tablet_manager.cc:1905] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.092916 958 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.093772 1178 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.093919 1178 ts_tablet_manager.cc:1397] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.003s
I20250112 20:51:38.094792 1178 raft_consensus.cc:357] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.095007 1178 raft_consensus.cc:383] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.095053 1178 raft_consensus.cc:738] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.097208 2238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Bootstrap starting.
I20250112 20:51:38.093976 1164 tablet_bootstrap.cc:654] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.100978 1178 consensus_queue.cc:260] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.102367 1178 ts_tablet_manager.cc:1428] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:38.103299 1178 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.104199 1164 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.104769 1164 ts_tablet_manager.cc:1397] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250112 20:51:38.106838 1164 raft_consensus.cc:357] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.111685 1164 raft_consensus.cc:383] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.111899 1164 raft_consensus.cc:738] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.112800 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 572172a274494e429b7e982815897e61 (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
I20250112 20:51:38.113183 1164 consensus_queue.cc:260] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.117947 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90f38fee7fd8445a98f6af6d299318bc" } modify_external_catalogs: true
I20250112 20:51:38.118216 1184 tablet_replica.cc:331] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.119571 821 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.120594 1184 raft_consensus.cc:2238] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.121762 1184 raft_consensus.cc:2267] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.122460 1178 tablet_bootstrap.cc:654] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.125901 1184 ts_tablet_manager.cc:1905] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.129101 1186 ts_tablet_manager.cc:1918] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.132004 1186 log.cc:1198] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:38.132436 1186 ts_tablet_manager.cc:1939] T 9f0092258e094a38ba7e223be03d4c9a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.132819 1163 tablet_bootstrap.cc:492] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:38.132946 1163 ts_tablet_manager.cc:1397] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.000s
I20250112 20:51:38.133425 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:38.133668 1186 tablet_replica.cc:331] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.133896 1186 raft_consensus.cc:2238] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:38.135026 1186 raft_consensus.cc:2267] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.134999 1163 raft_consensus.cc:357] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.135191 1163 raft_consensus.cc:383] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.135255 1163 raft_consensus.cc:738] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.135743 1163 consensus_queue.cc:260] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.136696 1186 ts_tablet_manager.cc:1905] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.140936 1163 ts_tablet_manager.cc:1428] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:38.142763 2059 raft_consensus.cc:491] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.142915 2059 raft_consensus.cc:513] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.143532 2059 leader_election.cc:290] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.146306 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.146668 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.146847 912 raft_consensus.cc:2463] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:38.147365 710 leader_election.cc:343] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.147552 710 leader_election.cc:304] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.148177 2212 raft_consensus.cc:2798] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.148259 2212 raft_consensus.cc:491] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.148332 2212 raft_consensus.cc:3054] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.151803 2212 raft_consensus.cc:513] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.151564 1164 ts_tablet_manager.cc:1428] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20250112 20:51:38.153534 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.154284 1164 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.154736 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:38.155306 912 raft_consensus.cc:3054] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:38.155988 710 leader_election.cc:343] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.158840 2140 raft_consensus.cc:491] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.158977 2140 raft_consensus.cc:513] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.159593 2140 leader_election.cc:290] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.160333 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.160547 910 raft_consensus.cc:2388] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:38.162298 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e390eb2e404250a8b6edba2ee4f6d6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.165380 1185 ts_tablet_manager.cc:1918] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.166100 1185 log.cc:1198] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:38.169652 1185 ts_tablet_manager.cc:1939] T 9f0092258e094a38ba7e223be03d4c9a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.170406 1164 tablet_bootstrap.cc:654] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.171618 1184 ts_tablet_manager.cc:1918] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.172390 1184 log.cc:1198] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d66c55c29de7458bb8ea69b826c567d7
W20250112 20:51:38.172813 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bb9c2e6f37f4184a106aed05bae7924 consensus state has no leader
I20250112 20:51:38.175357 1184 ts_tablet_manager.cc:1939] T d66c55c29de7458bb8ea69b826c567d7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.175550 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9f0092258e094a38ba7e223be03d4c9a (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
W20250112 20:51:38.177371 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 43693.3us
I20250112 20:51:38.152738 2212 leader_election.cc:290] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:38.178738 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8c5b10576bd34af5b02f8d8dcc3759ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c5b10576bd34af5b02f8d8dcc3759ab already in progress: creating tablet
I20250112 20:51:38.179212 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d66c55c29de7458bb8ea69b826c567d7 (table default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]) successfully deleted
I20250112 20:51:38.180133 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9f0092258e094a38ba7e223be03d4c9a (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
W20250112 20:51:38.180366 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet dc3a9740415140fe8864d54f477af954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc3a9740415140fe8864d54f477af954 already in progress: creating tablet
W20250112 20:51:38.181252 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b30a34c1abcb4f72a7f4f8a8f5be3720 already in progress: creating tablet
I20250112 20:51:38.180984 1184 tablet_replica.cc:331] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:38.181736 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 913149098822406e8d4ca5a440c9afd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913149098822406e8d4ca5a440c9afd7 already in progress: creating tablet
I20250112 20:51:38.182493 1184 raft_consensus.cc:2238] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.183293 1184 raft_consensus.cc:2267] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:38.184110 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8a11203ca15745c0a89a448e1cfc2cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a11203ca15745c0a89a448e1cfc2cc4 already in progress: creating tablet
W20250112 20:51:38.184357 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c5b10576bd34af5b02f8d8dcc3759ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c5b10576bd34af5b02f8d8dcc3759ab already in progress: creating tablet
I20250112 20:51:38.184068 1185 tablet_replica.cc:331] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.185572 1185 raft_consensus.cc:2238] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.184392 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fec3d1a320fe450dbab6cf50e1380d8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.186412 1185 raft_consensus.cc:2267] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.186681 1184 ts_tablet_manager.cc:1905] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.190876 912 raft_consensus.cc:2463] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:38.195495 1164 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.195706 1164 ts_tablet_manager.cc:1397] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.002s
I20250112 20:51:38.197196 1164 raft_consensus.cc:357] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.197475 1164 raft_consensus.cc:383] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.197549 1164 raft_consensus.cc:738] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.197799 2108 raft_consensus.cc:491] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.198019 710 leader_election.cc:304] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.197849 1164 consensus_queue.cc:260] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.198508 2059 raft_consensus.cc:2798] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.198619 2059 raft_consensus.cc:695] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:38.198841 2059 consensus_queue.cc:237] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.199123 1164 ts_tablet_manager.cc:1428] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:38.205013 1185 ts_tablet_manager.cc:1905] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.206302 2108 raft_consensus.cc:513] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.206938 2108 leader_election.cc:290] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.209800 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85897bfec1834c57956a23c5dfc2e853" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.210626 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85897bfec1834c57956a23c5dfc2e853" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.210830 774 raft_consensus.cc:2463] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.211643 847 leader_election.cc:304] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:38.212009 847 leader_election.cc:343] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.212453 2108 raft_consensus.cc:2798] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.212567 2108 raft_consensus.cc:491] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.212679 2108 raft_consensus.cc:3054] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.215176 2108 raft_consensus.cc:513] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.215876 2108 leader_election.cc:290] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.217418 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:38.218477 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]). Expected version 2 got 0
W20250112 20:51:38.224027 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a89187312df241109454241edcfd1335 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89187312df241109454241edcfd1335 already in progress: creating tablet
I20250112 20:51:38.224990 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85897bfec1834c57956a23c5dfc2e853" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.224956 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_11f04466bde745949068ba1eac6f68a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.225468 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11f04466bde745949068ba1eac6f68a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.226475 847 leader_election.cc:343] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.226876 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85897bfec1834c57956a23c5dfc2e853" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.227046 774 raft_consensus.cc:3054] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.228072 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_abecc332f4f147418a2479c3e0f768f5" } new_table_name: "default.Table_31c3d595c24b47dcb822727869bf8428" modify_external_catalogs: true
I20250112 20:51:38.229146 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a9962f17769a4deaab73b90205bb15bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.229624 774 raft_consensus.cc:2463] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:38.229641 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9962f17769a4deaab73b90205bb15bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.230163 1178 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.230285 1178 ts_tablet_manager.cc:1397] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.008s
I20250112 20:51:38.230186 1184 ts_tablet_manager.cc:1918] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:38.224061 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fec3d1a320fe450dbab6cf50e1380d8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.230633 1184 log.cc:1198] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/572172a274494e429b7e982815897e61
I20250112 20:51:38.231083 1178 raft_consensus.cc:357] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.231272 1178 raft_consensus.cc:383] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.231325 1178 raft_consensus.cc:738] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.232218 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2378bcfbef514d9a8e2bc42f74afada5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.232664 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2378bcfbef514d9a8e2bc42f74afada5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.234320 1186 ts_tablet_manager.cc:1918] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:38.234413 1186 log.cc:1198] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7293510f40804e2aabccf92986489dab
I20250112 20:51:38.234752 1186 ts_tablet_manager.cc:1939] T 7293510f40804e2aabccf92986489dab P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.232749 1184 ts_tablet_manager.cc:1939] T 572172a274494e429b7e982815897e61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.235412 1186 tablet_replica.cc:331] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.235595 1186 raft_consensus.cc:2238] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.231537 1178 consensus_queue.cc:260] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.237390 1017 tablet_service.cc:1467] Processing CreateTablet for tablet cbdc5c117ca9437091a1667642c7c846 (DEFAULT_TABLE table=default.Table_2a71d5cf4d36447da49bad300ee1b392 [id=c5fd2a74abcb4c8c938a5a3aacd7a427]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.237519 1178 ts_tablet_manager.cc:1428] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:38.237433 1185 ts_tablet_manager.cc:1918] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:38.237711 1185 log.cc:1198] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7293510f40804e2aabccf92986489dab
I20250112 20:51:38.239187 754 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea347fd2694016a3cd3721b0cab608 (DEFAULT_TABLE table=default.Table_a64bcccf91b84c25b6a7cbae25eedc8a [id=e6282890375942f3a96430d8a9730fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.239359 738 tablet_service.cc:1467] Processing CreateTablet for tablet cbdc5c117ca9437091a1667642c7c846 (DEFAULT_TABLE table=default.Table_2a71d5cf4d36447da49bad300ee1b392 [id=c5fd2a74abcb4c8c938a5a3aacd7a427]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.239763 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 49bfc4fbe78745049699c8383c724f38 (DEFAULT_TABLE table=default.Table_7f659b016002468f88844b86ed218e32 [id=03a3627293fe4ab7b982862f09ba143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.239804 847 leader_election.cc:304] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.239777 880 tablet_service.cc:1467] Processing CreateTablet for tablet cbdc5c117ca9437091a1667642c7c846 (DEFAULT_TABLE table=default.Table_2a71d5cf4d36447da49bad300ee1b392 [id=c5fd2a74abcb4c8c938a5a3aacd7a427]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.240111 737 tablet_service.cc:1467] Processing CreateTablet for tablet 49bfc4fbe78745049699c8383c724f38 (DEFAULT_TABLE table=default.Table_7f659b016002468f88844b86ed218e32 [id=03a3627293fe4ab7b982862f09ba143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.240262 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:38.240283 885 tablet_service.cc:1467] Processing CreateTablet for tablet 49bfc4fbe78745049699c8383c724f38 (DEFAULT_TABLE table=default.Table_7f659b016002468f88844b86ed218e32 [id=03a3627293fe4ab7b982862f09ba143b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.240449 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea347fd2694016a3cd3721b0cab608 (DEFAULT_TABLE table=default.Table_a64bcccf91b84c25b6a7cbae25eedc8a [id=e6282890375942f3a96430d8a9730fde]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:38.240509 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a89187312df241109454241edcfd1335 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89187312df241109454241edcfd1335 already in progress: creating tablet
W20250112 20:51:38.240711 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8a11203ca15745c0a89a448e1cfc2cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a11203ca15745c0a89a448e1cfc2cc4 already in progress: creating tablet
I20250112 20:51:38.240695 874 tablet_service.cc:1467] Processing CreateTablet for tablet d3ea347fd2694016a3cd3721b0cab608 (DEFAULT_TABLE table=default.Table_a64bcccf91b84c25b6a7cbae25eedc8a [id=e6282890375942f3a96430d8a9730fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.240747 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe (DEFAULT_TABLE table=default.Table_56d64a03c356468599c3310df24db01b [id=72c5896effc84cbeabce3c0ed125d3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.242384 1185 ts_tablet_manager.cc:1939] T 7293510f40804e2aabccf92986489dab P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.243353 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 924082716c1e4d9ea6fbe97ae93232bd
I20250112 20:51:38.243546 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3994d41a7ecd418a98a3c203add0d4d4
I20250112 20:51:38.244284 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7f164d4ff332493e885b6ef5272e508a" } new_table_name: "default.Table_b96c62442b9e4c6a8ee4b88a6a990e6f" modify_external_catalogs: true
I20250112 20:51:38.245234 1982 raft_consensus.cc:2798] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.245354 1982 raft_consensus.cc:695] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.245577 1982 consensus_queue.cc:237] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.245802 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 85897bfec1834c57956a23c5dfc2e853 (table default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7]). Expected version 1 got 0
I20250112 20:51:38.246083 446 catalog_manager.cc:5526] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:38.247232 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4303e4aec9694c49aca7c1532eaab3ed
I20250112 20:51:38.247637 2140 raft_consensus.cc:491] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.247848 1178 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.247774 2140 raft_consensus.cc:513] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.247867 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_be8270d1ca354e748dac253a4aeb09a3" } modify_external_catalogs: true
I20250112 20:51:38.248030 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_be8270d1ca354e748dac253a4aeb09a3" } modify_external_catalogs: true
I20250112 20:51:38.248387 2140 leader_election.cc:290] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:38.249333 983 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:38.249600 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cc39ddff5b22412fb084d77879afe4da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.250121 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc39ddff5b22412fb084d77879afe4da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.250808 983 leader_election.cc:343] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:38.250901 983 leader_election.cc:304] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:38.235733 1186 raft_consensus.cc:2267] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.251343 2140 raft_consensus.cc:2743] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:38.246443 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (table default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2]). Expected version 1 got 0
I20250112 20:51:38.251641 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 38cdb9a415f745cbb62f27763e490d5b (table default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7]). Expected version 1 got 0
I20250112 20:51:38.251871 1186 ts_tablet_manager.cc:1905] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.252167 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdc5c117ca9437091a1667642c7c846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.254549 1178 tablet_bootstrap.cc:654] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.255460 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bfc4fbe78745049699c8383c724f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.255443 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.255506 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.257244 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.259028 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.259380 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.259771 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.261087 1164 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.262419 1164 tablet_bootstrap.cc:654] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.263134 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ea347fd2694016a3cd3721b0cab608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.263258 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.265919 2108 raft_consensus.cc:491] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.266026 2108 raft_consensus.cc:513] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.266855 2108 leader_election.cc:290] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.269181 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ea347fd2694016a3cd3721b0cab608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.271692 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.271903 774 raft_consensus.cc:2463] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.272995 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdc5c117ca9437091a1667642c7c846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.276968 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bfc4fbe78745049699c8383c724f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.267592 1186 ts_tablet_manager.cc:1918] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.268488 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4303e4aec9694c49aca7c1532eaab3ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.269852 1178 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.280021 1178 ts_tablet_manager.cc:1397] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250112 20:51:38.280709 1178 raft_consensus.cc:357] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.280900 1178 raft_consensus.cc:383] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.280946 1178 raft_consensus.cc:738] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.281123 1178 consensus_queue.cc:260] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.281569 1178 ts_tablet_manager.cc:1428] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:38.281791 1178 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.272387 847 leader_election.cc:304] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.273136 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.282101 1186 log.cc:1198] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/48e390eb2e404250a8b6edba2ee4f6d6
I20250112 20:51:38.282150 2197 raft_consensus.cc:2798] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.282220 2197 raft_consensus.cc:491] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.282284 2197 raft_consensus.cc:3054] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.282405 1186 ts_tablet_manager.cc:1939] T 48e390eb2e404250a8b6edba2ee4f6d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.282382 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.282846 1186 tablet_replica.cc:331] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.283008 1186 raft_consensus.cc:2238] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.283135 1186 raft_consensus.cc:2267] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.284621 2197 raft_consensus.cc:513] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.286640 1186 ts_tablet_manager.cc:1905] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.287184 2197 leader_election.cc:290] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.288755 1185 tablet_replica.cc:331] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.289793 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bfc4fbe78745049699c8383c724f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.290098 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdc5c117ca9437091a1667642c7c846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.290709 1178 tablet_bootstrap.cc:654] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.292881 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902fc636a0494462ae1e73b0cefc5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:38.293431 847 leader_election.cc:343] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.294193 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.294358 774 raft_consensus.cc:3054] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.296172 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.296629 1184 tablet_replica.cc:331] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.296850 1184 raft_consensus.cc:2238] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.296869 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ea347fd2694016a3cd3721b0cab608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.297129 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 572172a274494e429b7e982815897e61 (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
I20250112 20:51:38.297426 774 raft_consensus.cc:2463] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.301048 1184 raft_consensus.cc:2267] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.302093 1184 ts_tablet_manager.cc:1905] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:38.302805 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e72a6c8e18934b789a048d5aaded121b consensus state has no leader
I20250112 20:51:38.306475 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 725e56fa8ac343e5b1ec1214da02b7d4
I20250112 20:51:38.306129 1185 raft_consensus.cc:2238] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:38.306524 1186 ts_tablet_manager.cc:1918] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.306614 1186 log.cc:1198] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/95d8e68746cf444ebac5e22728ab6f34
I20250112 20:51:38.306860 1186 ts_tablet_manager.cc:1939] T 95d8e68746cf444ebac5e22728ab6f34 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.307332 1186 tablet_replica.cc:331] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.307965 2212 raft_consensus.cc:491] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.308187 1185 pending_rounds.cc:62] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:38.308166 2212 raft_consensus.cc:513] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.309057 2212 leader_election.cc:290] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.308473 1185 pending_rounds.cc:69] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585040087113728 op_type: NO_OP noop_request { }
I20250112 20:51:38.309386 1185 raft_consensus.cc:2883] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:38.309806 1185 raft_consensus.cc:2267] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.310302 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b532068d2374b59a3c46525db7c50db" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.309803 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b532068d2374b59a3c46525db7c50db" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.311146 912 raft_consensus.cc:2463] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:38.311194 710 leader_election.cc:343] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.311585 710 leader_election.cc:304] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.312717 2059 raft_consensus.cc:2798] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.312793 2059 raft_consensus.cc:491] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.312909 2059 raft_consensus.cc:3054] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.313500 1185 ts_tablet_manager.cc:1905] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.314150 847 leader_election.cc:304] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.315421 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7293510f40804e2aabccf92986489dab (table default.Table_54cf793124084547a2343f0a71d88796 [id=86e79855b8c7432b8b460e4c16f3be75]) successfully deleted
I20250112 20:51:38.315451 2059 raft_consensus.cc:513] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.315902 2108 raft_consensus.cc:2798] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.315979 2059 leader_election.cc:290] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:38.316144 847 leader_election.cc:343] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.316985 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b532068d2374b59a3c46525db7c50db" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.316161 2108 raft_consensus.cc:695] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.317512 2108 consensus_queue.cc:237] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.317834 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b532068d2374b59a3c46525db7c50db" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:38.317994 912 raft_consensus.cc:3054] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.321107 912 raft_consensus.cc:2463] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:38.321475 710 leader_election.cc:304] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:38.321732 710 leader_election.cc:343] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.321974 2212 raft_consensus.cc:2798] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.322233 2212 raft_consensus.cc:695] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:38.322430 2212 consensus_queue.cc:237] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.325735 1186 raft_consensus.cc:2238] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:38.327308 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d72ae28879dc45429de2ab0d66276255 consensus state has no leader
I20250112 20:51:38.328107 1186 raft_consensus.cc:2267] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.321527 1163 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:38.332171 1186 ts_tablet_manager.cc:1905] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.332306 1163 tablet_bootstrap.cc:654] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.326418 1178 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.332911 1178 ts_tablet_manager.cc:1397] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20250112 20:51:38.333544 1178 raft_consensus.cc:357] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.333734 1178 raft_consensus.cc:383] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.333796 1178 raft_consensus.cc:738] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.334049 1178 consensus_queue.cc:260] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.334612 1178 ts_tablet_manager.cc:1428] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:38.334873 1178 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.336249 1178 tablet_bootstrap.cc:654] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.338196 1164 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.338317 1164 ts_tablet_manager.cc:1397] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.003s
I20250112 20:51:38.339094 1164 raft_consensus.cc:357] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.339146 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c0e48e09ccab48a9a2cf6aa4212f1658" } modify_external_catalogs: true
I20250112 20:51:38.339282 1164 raft_consensus.cc:383] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.339339 1164 raft_consensus.cc:738] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.339321 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c0e48e09ccab48a9a2cf6aa4212f1658" } modify_external_catalogs: true
I20250112 20:51:38.339531 1164 consensus_queue.cc:260] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.340044 1164 ts_tablet_manager.cc:1428] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:38.340299 1164 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.340577 452 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:38.341120 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_93a664f616254b98a1914cedad79b639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.341660 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93a664f616254b98a1914cedad79b639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.344321 1186 ts_tablet_manager.cc:1918] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.351055 1164 tablet_bootstrap.cc:654] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.346859 1178 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.354157 1178 ts_tablet_manager.cc:1397] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250112 20:51:38.354784 1178 raft_consensus.cc:357] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.354976 1178 raft_consensus.cc:383] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.355044 1178 raft_consensus.cc:738] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.354620 1184 ts_tablet_manager.cc:1918] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.355264 1178 consensus_queue.cc:260] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.355682 1178 ts_tablet_manager.cc:1428] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:38.355928 1178 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.356380 1184 log.cc:1198] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9f0092258e094a38ba7e223be03d4c9a
I20250112 20:51:38.357102 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 65784493c6ef417993b1f6371cab47d6 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:38.357331 1178 tablet_bootstrap.cc:654] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.357686 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 48e390eb2e404250a8b6edba2ee4f6d6 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
I20250112 20:51:38.358035 1186 log.cc:1198] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d66c55c29de7458bb8ea69b826c567d7
I20250112 20:51:38.358374 1186 ts_tablet_manager.cc:1939] T d66c55c29de7458bb8ea69b826c567d7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.359046 1184 ts_tablet_manager.cc:1939] T 9f0092258e094a38ba7e223be03d4c9a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.359364 1186 tablet_replica.cc:331] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.359565 1186 raft_consensus.cc:2238] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.359807 1186 raft_consensus.cc:2267] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.361301 1186 ts_tablet_manager.cc:1905] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.359709 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7d7f37dbdb4c43eab0f6d7bde31ec32e" } new_table_name: "default.Table_cba467c888b84da6964f049247721310" modify_external_catalogs: true
I20250112 20:51:38.365491 1184 tablet_replica.cc:331] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.360090 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_230aea5bb3074c90aa9e772ab27c1bce" } new_table_name: "default.Table_a01cef1a596c47309a0b659209428d4b" modify_external_catalogs: true
I20250112 20:51:38.366497 1184 raft_consensus.cc:2238] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.367126 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 95d8e68746cf444ebac5e22728ab6f34 (table default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]) successfully deleted
I20250112 20:51:38.366961 1185 ts_tablet_manager.cc:1918] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:38.367408 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 913149098822406e8d4ca5a440c9afd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913149098822406e8d4ca5a440c9afd7 already in progress: creating tablet
I20250112 20:51:38.367502 1184 raft_consensus.cc:2267] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.367712 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d66c55c29de7458bb8ea69b826c567d7 (table default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]) successfully deleted
I20250112 20:51:38.367770 1185 log.cc:1198] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/48e390eb2e404250a8b6edba2ee4f6d6
W20250112 20:51:38.367955 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f8d521e6c8cd461887f93f395c27b89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d521e6c8cd461887f93f395c27b89e already in progress: creating tablet
W20250112 20:51:38.368157 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f425779195594599a3ab8c4dc5bee7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f425779195594599a3ab8c4dc5bee7f9 already in progress: creating tablet
I20250112 20:51:38.368453 1178 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.368604 1178 ts_tablet_manager.cc:1397] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20250112 20:51:38.369388 1178 raft_consensus.cc:357] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.369654 1178 raft_consensus.cc:383] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.369732 1178 raft_consensus.cc:738] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.369985 1178 consensus_queue.cc:260] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.370945 1178 ts_tablet_manager.cc:1428] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:38.371259 1178 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.373030 1178 tablet_bootstrap.cc:654] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.375078 1185 ts_tablet_manager.cc:1939] T 48e390eb2e404250a8b6edba2ee4f6d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.375805 1184 ts_tablet_manager.cc:1905] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.383651 1164 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.383848 1164 ts_tablet_manager.cc:1397] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20250112 20:51:38.384660 1164 raft_consensus.cc:357] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.384974 1164 raft_consensus.cc:383] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.385102 1164 raft_consensus.cc:738] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.385488 1164 consensus_queue.cc:260] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.386488 1164 ts_tablet_manager.cc:1428] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:38.388927 2208 raft_consensus.cc:491] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.389052 2208 raft_consensus.cc:513] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.389602 2208 leader_election.cc:290] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.393877 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.394322 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 903027fe49e34a308108356ba89a9c3a
I20250112 20:51:38.396039 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.396235 912 raft_consensus.cc:2463] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:38.396732 710 leader_election.cc:343] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.398185 710 leader_election.cc:304] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.398557 2208 raft_consensus.cc:2798] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.398648 2208 raft_consensus.cc:491] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.398702 2208 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.405916 2208 raft_consensus.cc:513] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.406634 2208 leader_election.cc:290] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.407433 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:38.407608 912 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.407925 1185 tablet_replica.cc:331] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.408119 1185 raft_consensus.cc:2238] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.408257 1185 raft_consensus.cc:2267] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.408731 1185 ts_tablet_manager.cc:1905] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.410250 912 raft_consensus.cc:2463] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:38.410706 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.411438 710 leader_election.cc:304] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:38.411728 710 leader_election.cc:343] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.413230 1164 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.414829 1164 tablet_bootstrap.cc:654] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.417368 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.417769 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.417989 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.418661 880 tablet_service.cc:1467] Processing CreateTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe (DEFAULT_TABLE table=default.Table_56d64a03c356468599c3310df24db01b [id=72c5896effc84cbeabce3c0ed125d3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.418643 738 tablet_service.cc:1467] Processing CreateTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe (DEFAULT_TABLE table=default.Table_56d64a03c356468599c3310df24db01b [id=72c5896effc84cbeabce3c0ed125d3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.418982 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902fc636a0494462ae1e73b0cefc5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.418982 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902fc636a0494462ae1e73b0cefc5dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.422698 2208 raft_consensus.cc:2798] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.422909 2208 raft_consensus.cc:695] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:38.423115 2208 consensus_queue.cc:237] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.430752 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a218ef5825424b1b84c55b38b9b9023d (table default.Table_df950a6830fa45b1a8f21feda53ea0ea [id=05211d8527144710860dc2aca1ac76a3]). Expected version 1 got 0
I20250112 20:51:38.431056 449 catalog_manager.cc:5526] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:38.431597 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 85897bfec1834c57956a23c5dfc2e853 (table default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7]). Expected version 1 got 0
I20250112 20:51:38.431427 1185 ts_tablet_manager.cc:1918] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.431897 1185 log.cc:1198] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8d08048c402345a686d1e3fb3039bbc1
I20250112 20:51:38.431895 449 catalog_manager.cc:5526] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:38.432718 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 38cdb9a415f745cbb62f27763e490d5b (table default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7]). Expected version 1 got 0
I20250112 20:51:38.433109 449 catalog_manager.cc:5526] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:38.433519 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:38.435127 1185 ts_tablet_manager.cc:1939] T 8d08048c402345a686d1e3fb3039bbc1 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.438315 1184 ts_tablet_manager.cc:1918] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.440742 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 26c3f3095603490f8861b1b26d5fe78d (DEFAULT_TABLE table=default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41 [id=9ca6b95a0def4511bcc96d380c2ed0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.441341 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c3f3095603490f8861b1b26d5fe78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.444386 1178 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.444492 1178 ts_tablet_manager.cc:1397] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.001s
I20250112 20:51:38.445133 1178 raft_consensus.cc:357] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.445333 1178 raft_consensus.cc:383] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.445382 1178 raft_consensus.cc:738] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.445564 1178 consensus_queue.cc:260] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.446062 1178 ts_tablet_manager.cc:1428] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:38.446182 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1a505986bc6146829cbafd0de6682d93 (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.446182 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1a505986bc6146829cbafd0de6682d93 (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.446542 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a505986bc6146829cbafd0de6682d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.447304 1163 tablet_bootstrap.cc:492] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:38.447436 1163 ts_tablet_manager.cc:1397] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.000s
I20250112 20:51:38.448215 1163 raft_consensus.cc:357] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.448388 1163 raft_consensus.cc:383] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.448441 1163 raft_consensus.cc:738] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.449009 880 tablet_service.cc:1467] Processing CreateTablet for tablet 26c3f3095603490f8861b1b26d5fe78d (DEFAULT_TABLE table=default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41 [id=9ca6b95a0def4511bcc96d380c2ed0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.449071 738 tablet_service.cc:1467] Processing CreateTablet for tablet 26c3f3095603490f8861b1b26d5fe78d (DEFAULT_TABLE table=default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41 [id=9ca6b95a0def4511bcc96d380c2ed0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.449452 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c3f3095603490f8861b1b26d5fe78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.453083 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a505986bc6146829cbafd0de6682d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.456636 1178 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.459491 1178 tablet_bootstrap.cc:654] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.460145 1184 log.cc:1198] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/54de54eabe5946058d1940b79f5fb832
I20250112 20:51:38.460464 1184 ts_tablet_manager.cc:1939] T 54de54eabe5946058d1940b79f5fb832 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.460845 1048 raft_consensus.cc:3054] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.461077 1185 tablet_replica.cc:331] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.461253 1185 raft_consensus.cc:2238] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.461400 1185 raft_consensus.cc:2267] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.463148 1185 ts_tablet_manager.cc:1905] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.464707 1184 tablet_replica.cc:331] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.464934 1184 raft_consensus.cc:2238] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.465087 1184 raft_consensus.cc:2267] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.465318 2208 consensus_queue.cc:1035] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.465564 1184 ts_tablet_manager.cc:1905] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.465798 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c3f3095603490f8861b1b26d5fe78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.471551 2108 raft_consensus.cc:491] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.471731 2108 raft_consensus.cc:513] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.472494 2108 leader_election.cc:290] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.472775 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310e4135c2b424fb006a38a60e41066" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:38.473637 847 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.473608 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310e4135c2b424fb006a38a60e41066" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.478179 1164 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.478212 1185 ts_tablet_manager.cc:1918] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.478499 1164 ts_tablet_manager.cc:1397] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.006s
I20250112 20:51:38.480015 1164 raft_consensus.cc:357] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.480382 1164 raft_consensus.cc:383] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.480487 2208 consensus_queue.cc:1035] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.480547 1164 raft_consensus.cc:738] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.481092 1164 consensus_queue.cc:260] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.481735 1185 log.cc:1198] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d66c55c29de7458bb8ea69b826c567d7
I20250112 20:51:38.482811 1184 ts_tablet_manager.cc:1918] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.482908 1164 ts_tablet_manager.cc:1428] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:38.483044 1184 log.cc:1198] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4303e4aec9694c49aca7c1532eaab3ed
I20250112 20:51:38.483520 1185 ts_tablet_manager.cc:1939] T d66c55c29de7458bb8ea69b826c567d7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.483695 1184 ts_tablet_manager.cc:1939] T 4303e4aec9694c49aca7c1532eaab3ed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:38.483848 847 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.484227 1164 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.484406 847 leader_election.cc:304] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:38.485255 2108 raft_consensus.cc:2743] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:38.489564 1185 tablet_replica.cc:331] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.490804 1164 tablet_bootstrap.cc:654] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.490981 1185 raft_consensus.cc:2238] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.491866 1185 raft_consensus.cc:2267] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.492396 1185 ts_tablet_manager.cc:1905] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.448689 1163 consensus_queue.cc:260] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.498911 2108 raft_consensus.cc:491] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.499064 2108 raft_consensus.cc:513] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.499658 1185 ts_tablet_manager.cc:1918] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.499665 2108 leader_election.cc:290] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.499753 1185 log.cc:1198] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1d60604713524dba8f3a42c4f3bc7300
I20250112 20:51:38.500003 1185 ts_tablet_manager.cc:1939] T 1d60604713524dba8f3a42c4f3bc7300 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.498512 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1a505986bc6146829cbafd0de6682d93 (DEFAULT_TABLE table=default.Table_0bb556ec81044d1f8296628212fc4875 [id=43e0a95b862b4686b97704dd3c7a551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.500842 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0a808d431a43fb9299fec1dcf6d3b9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.501021 774 raft_consensus.cc:2463] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.501192 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a505986bc6146829cbafd0de6682d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.501757 847 leader_election.cc:304] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.504977 1164 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.505156 1164 ts_tablet_manager.cc:1397] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20250112 20:51:38.505924 1164 raft_consensus.cc:357] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.506160 1164 raft_consensus.cc:383] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.506249 1164 raft_consensus.cc:738] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.506515 1164 consensus_queue.cc:260] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.507442 1164 ts_tablet_manager.cc:1428] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:38.507712 1164 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.508374 2108 raft_consensus.cc:2798] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.508476 2108 raft_consensus.cc:491] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.508562 2108 raft_consensus.cc:3054] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.510004 1164 tablet_bootstrap.cc:654] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.511350 2108 raft_consensus.cc:513] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.511775 1185 tablet_replica.cc:331] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.512017 1185 raft_consensus.cc:2238] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.512209 1185 raft_consensus.cc:2267] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.512382 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0a808d431a43fb9299fec1dcf6d3b9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.512440 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0a808d431a43fb9299fec1dcf6d3b9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.512569 774 raft_consensus.cc:3054] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.500453 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0a808d431a43fb9299fec1dcf6d3b9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:38.512850 847 leader_election.cc:343] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.512058 2108 leader_election.cc:290] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:38.513135 847 leader_election.cc:343] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.515020 774 raft_consensus.cc:2463] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.515776 1185 ts_tablet_manager.cc:1905] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.516600 847 leader_election.cc:304] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.517163 2076 raft_consensus.cc:2798] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.517282 2076 raft_consensus.cc:695] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.517495 2076 consensus_queue.cc:237] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.523221 1163 ts_tablet_manager.cc:1428] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.076s user 0.000s sys 0.002s
I20250112 20:51:38.525189 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.526046 1164 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.526242 1164 ts_tablet_manager.cc:1397] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250112 20:51:38.527002 1164 raft_consensus.cc:357] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.527257 1164 raft_consensus.cc:383] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.527359 1164 raft_consensus.cc:738] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.527583 1164 consensus_queue.cc:260] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.527706 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 19fc10bb8452482dabe12950fd0ea55f (DEFAULT_TABLE table=default.Table_2c470161ee88485e98f09828f7e9f22c [id=ca65adc457fb4079af6c09d532d43f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.528272 1164 ts_tablet_manager.cc:1428] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:38.528563 1164 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.530272 1164 tablet_bootstrap.cc:654] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.530508 1185 ts_tablet_manager.cc:1918] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.531265 1185 log.cc:1198] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4303e4aec9694c49aca7c1532eaab3ed
I20250112 20:51:38.531606 1185 ts_tablet_manager.cc:1939] T 4303e4aec9694c49aca7c1532eaab3ed P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.537546 1164 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.537721 1164 ts_tablet_manager.cc:1397] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250112 20:51:38.539026 885 tablet_service.cc:1467] Processing CreateTablet for tablet 19fc10bb8452482dabe12950fd0ea55f (DEFAULT_TABLE table=default.Table_2c470161ee88485e98f09828f7e9f22c [id=ca65adc457fb4079af6c09d532d43f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.539424 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fc10bb8452482dabe12950fd0ea55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.539381 737 tablet_service.cc:1467] Processing CreateTablet for tablet 19fc10bb8452482dabe12950fd0ea55f (DEFAULT_TABLE table=default.Table_2c470161ee88485e98f09828f7e9f22c [id=ca65adc457fb4079af6c09d532d43f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.539680 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fc10bb8452482dabe12950fd0ea55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.544680 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b (DEFAULT_TABLE table=default.Table_d1dcab09931f446da585ba49901bb41d [id=03fef3e9bc4745b9935413c278c6e794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.550407 879 tablet_service.cc:1467] Processing CreateTablet for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b (DEFAULT_TABLE table=default.Table_d1dcab09931f446da585ba49901bb41d [id=03fef3e9bc4745b9935413c278c6e794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.550653 754 tablet_service.cc:1467] Processing CreateTablet for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b (DEFAULT_TABLE table=default.Table_d1dcab09931f446da585ba49901bb41d [id=03fef3e9bc4745b9935413c278c6e794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.550844 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.550993 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.557168 2108 raft_consensus.cc:491] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.557291 2108 raft_consensus.cc:513] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.558576 2108 leader_election.cc:290] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.558640 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a218ef5825424b1b84c55b38b9b9023d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:38.558827 774 raft_consensus.cc:2463] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:38.558810 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a218ef5825424b1b84c55b38b9b9023d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.559206 847 leader_election.cc:304] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:38.559697 847 leader_election.cc:343] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.560027 2108 raft_consensus.cc:2798] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.560111 2108 raft_consensus.cc:491] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.560184 2108 raft_consensus.cc:3054] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.561631 1164 raft_consensus.cc:357] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.562007 1164 raft_consensus.cc:383] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.562119 1164 raft_consensus.cc:738] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.562353 1164 consensus_queue.cc:260] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.562765 2108 raft_consensus.cc:513] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.563071 1164 ts_tablet_manager.cc:1428] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250112 20:51:38.563362 1164 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.563684 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a218ef5825424b1b84c55b38b9b9023d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:38.564114 847 leader_election.cc:343] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.564496 1163 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:38.565201 1164 tablet_bootstrap.cc:654] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.566892 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb (DEFAULT_TABLE table=default.Table_917e8a15aa6b4cc28455521407819e42 [id=2a2fd503993447fdae883ffdf3c246e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:38.568348 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0fd4177f9394a5a82d4f5e02369b3b6 consensus state has no leader
W20250112 20:51:38.568869 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaa091fe2d3c4f72815f228ac3e39128 consensus state has no leader
I20250112 20:51:38.569039 737 tablet_service.cc:1467] Processing CreateTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb (DEFAULT_TABLE table=default.Table_917e8a15aa6b4cc28455521407819e42 [id=2a2fd503993447fdae883ffdf3c246e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.569383 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c36c1a2e7b42058f74a29a7c0d41eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.570037 879 tablet_service.cc:1467] Processing CreateTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb (DEFAULT_TABLE table=default.Table_917e8a15aa6b4cc28455521407819e42 [id=2a2fd503993447fdae883ffdf3c246e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.570386 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c36c1a2e7b42058f74a29a7c0d41eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.577698 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.578770 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e390eb2e404250a8b6edba2ee4f6d6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8229e4ef771b4032a4bfae3190a851e3) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.579192 737 tablet_service.cc:1467] Processing CreateTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.579629 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9498a04f20dd4cf4a283f17d37ca49ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.582042 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19752a4e5bc94b5bbbe1f16aa47c4d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.585006 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c36c1a2e7b42058f74a29a7c0d41eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.587514 1164 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.587702 1164 ts_tablet_manager.cc:1397] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20250112 20:51:38.588609 1164 raft_consensus.cc:357] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.589265 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9498a04f20dd4cf4a283f17d37ca49ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.588982 1164 raft_consensus.cc:383] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.589499 1164 raft_consensus.cc:738] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.589813 1164 consensus_queue.cc:260] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.590487 1164 ts_tablet_manager.cc:1428] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:38.590796 1164 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.592484 1164 tablet_bootstrap.cc:654] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.596873 1184 tablet_replica.cc:331] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.597141 1184 raft_consensus.cc:2238] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.597329 1184 raft_consensus.cc:2267] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.598280 1184 ts_tablet_manager.cc:1905] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.604871 1184 ts_tablet_manager.cc:1918] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.605043 1184 log.cc:1198] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/48e390eb2e404250a8b6edba2ee4f6d6
I20250112 20:51:38.605443 1184 ts_tablet_manager.cc:1939] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.610375 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fc10bb8452482dabe12950fd0ea55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.610848 1164 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.611095 1164 ts_tablet_manager.cc:1397] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20250112 20:51:38.612258 1164 raft_consensus.cc:357] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.612571 1164 raft_consensus.cc:383] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.612717 1164 raft_consensus.cc:738] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.613186 1164 consensus_queue.cc:260] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:38.614542 983 leader_election.cc:343] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: cannot vote while shut down
I20250112 20:51:38.614679 983 leader_election.cc:304] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:38.614849 983 raft_consensus.cc:2681] T 48e390eb2e404250a8b6edba2ee4f6d6 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:38.614588 1164 ts_tablet_manager.cc:1428] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:38.616220 1164 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.619403 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9f0092258e094a38ba7e223be03d4c9a (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
W20250112 20:51:38.619681 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f8d521e6c8cd461887f93f395c27b89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d521e6c8cd461887f93f395c27b89e already in progress: creating tablet
I20250112 20:51:38.620007 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 54de54eabe5946058d1940b79f5fb832 (table default.Table_f49b62d82ca74b20896a1eac33adf71a [id=cd6261ae00c342b58d92a875fb854b5a]) successfully deleted
W20250112 20:51:38.620325 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f425779195594599a3ab8c4dc5bee7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f425779195594599a3ab8c4dc5bee7f9 already in progress: creating tablet
W20250112 20:51:38.620573 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet fe1dd81dcdc94420bb12a34c1167da8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe1dd81dcdc94420bb12a34c1167da8a already in progress: creating tablet
W20250112 20:51:38.620880 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 924082716c1e4d9ea6fbe97ae93232bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924082716c1e4d9ea6fbe97ae93232bd already in progress: creating tablet
W20250112 20:51:38.621317 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3994d41a7ecd418a98a3c203add0d4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3994d41a7ecd418a98a3c203add0d4d4 already in progress: creating tablet
I20250112 20:51:38.621542 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]) successfully deleted
W20250112 20:51:38.621798 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 725e56fa8ac343e5b1ec1214da02b7d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725e56fa8ac343e5b1ec1214da02b7d4 already in progress: creating tablet
I20250112 20:51:38.622139 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 48e390eb2e404250a8b6edba2ee4f6d6 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
I20250112 20:51:38.623040 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.623327 1185 tablet_replica.cc:331] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:38.623523 1185 raft_consensus.cc:2238] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:38.625370 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.623730 1164 tablet_bootstrap.cc:654] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.630398 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 38cdb9a415f745cbb62f27763e490d5b (table default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7]). Expected version 1 got 0
I20250112 20:51:38.630661 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:38.630847 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:38.631035 454 catalog_manager.cc:5526] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:38.623778 1185 pending_rounds.cc:62] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:38.632019 1185 pending_rounds.cc:69] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585042730323968 op_type: NO_OP noop_request { }
I20250112 20:51:38.632162 1185 raft_consensus.cc:2883] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:38.632266 1185 raft_consensus.cc:2267] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.625036 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:38.633127 1185 ts_tablet_manager.cc:1905] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.625624 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.636446 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.636876 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9498a04f20dd4cf4a283f17d37ca49ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.640148 1185 ts_tablet_manager.cc:1918] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.640268 1185 log.cc:1198] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3b532068d2374b59a3c46525db7c50db
I20250112 20:51:38.640614 1185 ts_tablet_manager.cc:1939] T 3b532068d2374b59a3c46525db7c50db P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:38.644004 738 tablet_service.cc:1467] Processing CreateTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.644454 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438e2c14353c4a8cb7242d75bb3ba673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.646029 454 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:38.652199 2108 leader_election.cc:290] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:38.652735 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a218ef5825424b1b84c55b38b9b9023d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:38.652931 775 raft_consensus.cc:3054] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.655966 775 raft_consensus.cc:2463] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:38.656558 847 leader_election.cc:304] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.656987 1163 tablet_bootstrap.cc:654] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.657910 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.658473 2108 raft_consensus.cc:2798] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.658531 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438e2c14353c4a8cb7242d75bb3ba673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.659711 958 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.660943 1186 ts_tablet_manager.cc:1918] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.661051 1186 log.cc:1198] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3b532068d2374b59a3c46525db7c50db
I20250112 20:51:38.661441 1186 ts_tablet_manager.cc:1939] T 3b532068d2374b59a3c46525db7c50db P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.662292 1186 tablet_replica.cc:331] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.662483 1186 raft_consensus.cc:2238] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:38.662832 1186 pending_rounds.cc:62] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:38.662894 1186 pending_rounds.cc:69] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585041556819968 op_type: NO_OP noop_request { }
I20250112 20:51:38.662941 912 raft_consensus.cc:1270] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Refusing update from remote peer aefe1db8b19244d891b6bcd8971e9b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:38.663530 2276 consensus_queue.cc:1035] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.665843 1178 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.666036 1178 ts_tablet_manager.cc:1397] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.209s user 0.011s sys 0.039s
W20250112 20:51:38.666575 710 consensus_peers.cc:487] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:38.669457 1178 raft_consensus.cc:357] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.669731 1178 raft_consensus.cc:383] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.669855 1178 raft_consensus.cc:738] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.670130 1178 consensus_queue.cc:260] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.670895 1178 ts_tablet_manager.cc:1428] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:38.671283 1178 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.673323 880 tablet_service.cc:1467] Processing CreateTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.673893 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438e2c14353c4a8cb7242d75bb3ba673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.674705 1178 tablet_bootstrap.cc:654] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.677677 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
W20250112 20:51:38.680255 847 consensus_peers.cc:487] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:38.663061 2108 raft_consensus.cc:695] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:38.680589 775 raft_consensus.cc:1270] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:38.680238 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d66c55c29de7458bb8ea69b826c567d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 438e2c14353c4a8cb7242d75bb3ba673) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.681092 1164 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.681164 2076 consensus_queue.cc:1035] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.681376 1164 ts_tablet_manager.cc:1397] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.007s
I20250112 20:51:38.683096 1164 raft_consensus.cc:357] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.683477 1164 raft_consensus.cc:383] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.688207 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3b532068d2374b59a3c46525db7c50db (table default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]) successfully deleted
W20250112 20:51:38.688539 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet fe1dd81dcdc94420bb12a34c1167da8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe1dd81dcdc94420bb12a34c1167da8a already in progress: creating tablet
W20250112 20:51:38.688995 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 924082716c1e4d9ea6fbe97ae93232bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924082716c1e4d9ea6fbe97ae93232bd already in progress: creating tablet
I20250112 20:51:38.689878 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e236e2ca4b654f28a0175e3f7041d249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.689973 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_088aef221fd440b1a66db4f07efa4232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.690536 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9b0e136fb45d404eb7351439ce056024"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.690661 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_088aef221fd440b1a66db4f07efa4232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.691051 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b0e136fb45d404eb7351439ce056024 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.690060 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9a9d57f9339042c0bdc18fd15adae7a3" } new_table_name: "default.Table_3886e5ca0a0b4c71ac1ade9fcf9d8ab2" modify_external_catalogs: true
I20250112 20:51:38.683658 1164 raft_consensus.cc:738] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.688936 2276 raft_consensus.cc:491] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.692989 2276 raft_consensus.cc:513] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:38.693115 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e236e2ca4b654f28a0175e3f7041d249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.693572 2276 leader_election.cc:290] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.680620 2108 consensus_queue.cc:237] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.694200 1164 consensus_queue.cc:260] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.696882 1164 ts_tablet_manager.cc:1428] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20250112 20:51:38.697753 1164 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.699303 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b932b7e1e34d4ab18aae40047dc038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:38.700701 710 leader_election.cc:343] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.701222 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b932b7e1e34d4ab18aae40047dc038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.701464 912 raft_consensus.cc:2463] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:38.701933 710 leader_election.cc:304] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.702284 2276 raft_consensus.cc:2798] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.702387 2276 raft_consensus.cc:491] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.702481 2276 raft_consensus.cc:3054] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.705174 2276 raft_consensus.cc:513] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.706184 2276 leader_election.cc:290] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.710011 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b932b7e1e34d4ab18aae40047dc038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:38.710690 710 leader_election.cc:343] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.711035 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b932b7e1e34d4ab18aae40047dc038" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:38.711300 912 raft_consensus.cc:3054] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.714533 912 raft_consensus.cc:2463] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:38.714609 1164 tablet_bootstrap.cc:654] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.715003 710 leader_election.cc:304] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:38.715498 2276 raft_consensus.cc:2798] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:38.715644 2276 raft_consensus.cc:695] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:38.715893 2276 consensus_queue.cc:237] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:38.724267 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8098069639a2435ab500908db9ecd12a consensus state has no leader
I20250112 20:51:38.662992 1186 raft_consensus.cc:2883] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:38.733531 1186 raft_consensus.cc:2267] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.734679 1186 ts_tablet_manager.cc:1905] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.736212 1164 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.736510 1164 ts_tablet_manager.cc:1397] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.003s
I20250112 20:51:38.737653 1164 raft_consensus.cc:357] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.738022 1164 raft_consensus.cc:383] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.738309 1164 raft_consensus.cc:738] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.739362 1164 consensus_queue.cc:260] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.741339 1164 ts_tablet_manager.cc:1428] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:38.742480 1164 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.747138 2294 tablet.cc:1721] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:38.752584 1164 tablet_bootstrap.cc:654] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.754087 1163 tablet_bootstrap.cc:492] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:38.754256 1163 ts_tablet_manager.cc:1397] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.190s user 0.007s sys 0.007s
I20250112 20:51:38.754988 1163 raft_consensus.cc:357] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.755205 1163 raft_consensus.cc:383] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.755301 1163 raft_consensus.cc:738] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.757337 1163 consensus_queue.cc:260] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:38.761775 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:38.764990 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e07c52b8ba4040188dcdd21d2ffa00a6" } modify_external_catalogs: true
I20250112 20:51:38.765223 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e07c52b8ba4040188dcdd21d2ffa00a6" } modify_external_catalogs: true
I20250112 20:51:38.767975 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_455466a0838742f3a2cc35cdab729433" } modify_external_catalogs: true
I20250112 20:51:38.768187 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_455466a0838742f3a2cc35cdab729433" } modify_external_catalogs: true
I20250112 20:51:38.769827 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86bf39762f4f4650aa99eafae2452344" } modify_external_catalogs: true
I20250112 20:51:38.770063 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86bf39762f4f4650aa99eafae2452344" } modify_external_catalogs: true
I20250112 20:51:38.770038 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0e3604a7e91842a99e0046e67f3cb0cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.770622 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e3604a7e91842a99e0046e67f3cb0cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.772871 1178 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.773005 1178 ts_tablet_manager.cc:1397] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.102s user 0.007s sys 0.013s
I20250112 20:51:38.773736 1178 raft_consensus.cc:357] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.773908 1178 raft_consensus.cc:383] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.773965 1178 raft_consensus.cc:738] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.774335 1178 consensus_queue.cc:260] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.774971 1178 ts_tablet_manager.cc:1428] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:38.774874 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_073ac55e6a934b0b943cfb161a831bb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.775501 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073ac55e6a934b0b943cfb161a831bb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.776001 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0140e608c30d4711841b87316d32f6c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.776459 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0140e608c30d4711841b87316d32f6c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.777803 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dcdefc016f7546e39891ebe15594cba1" } new_table_name: "default.Table_bc09e8e983ec49c2b224a4b5273bacc5" modify_external_catalogs: true
I20250112 20:51:38.779203 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1a61b32d430d45e7875d19a9905ef7fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:38.779629 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a61b32d430d45e7875d19a9905ef7fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:38.780958 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d410c058166640f08a520ceaec274a90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:38.781215 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 48e390eb2e404250a8b6edba2ee4f6d6 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
W20250112 20:51:38.781448 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d410c058166640f08a520ceaec274a90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:38.781497 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 913149098822406e8d4ca5a440c9afd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913149098822406e8d4ca5a440c9afd7 already in progress: creating tablet
I20250112 20:51:38.781889 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8d08048c402345a686d1e3fb3039bbc1 (table default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]) successfully deleted
I20250112 20:51:38.782078 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d66c55c29de7458bb8ea69b826c567d7 (table default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]) successfully deleted
W20250112 20:51:38.782258 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f8d521e6c8cd461887f93f395c27b89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d521e6c8cd461887f93f395c27b89e already in progress: creating tablet
I20250112 20:51:38.781323 2294 tablet.cc:1721] T 38cdb9a415f745cbb62f27763e490d5b P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:38.782428 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1d60604713524dba8f3a42c4f3bc7300 (table default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]) successfully deleted
W20250112 20:51:38.782608 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f425779195594599a3ab8c4dc5bee7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f425779195594599a3ab8c4dc5bee7f9 already in progress: creating tablet
W20250112 20:51:38.782809 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet fe1dd81dcdc94420bb12a34c1167da8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe1dd81dcdc94420bb12a34c1167da8a already in progress: creating tablet
W20250112 20:51:38.782995 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 924082716c1e4d9ea6fbe97ae93232bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924082716c1e4d9ea6fbe97ae93232bd already in progress: creating tablet
I20250112 20:51:38.783624 2238 log.cc:826] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:38.783742 2297 tablet.cc:1721] T f5aca1ff0fd541bfa96bf7f279fa4e2d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:38.785012 2293 tablet.cc:1721] T 38cdb9a415f745cbb62f27763e490d5b P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:38.797427 1186 ts_tablet_manager.cc:1918] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:38.797602 1186 log.cc:1198] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4303e4aec9694c49aca7c1532eaab3ed
I20250112 20:51:38.797979 1186 ts_tablet_manager.cc:1939] T 4303e4aec9694c49aca7c1532eaab3ed P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:38.798446 1164 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.799510 1164 ts_tablet_manager.cc:1397] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.002s
I20250112 20:51:38.806257 1178 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.807098 1164 raft_consensus.cc:357] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.808966 1164 raft_consensus.cc:383] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.809813 1164 raft_consensus.cc:738] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.810393 1178 tablet_bootstrap.cc:654] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:38.811779 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3994d41a7ecd418a98a3c203add0d4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3994d41a7ecd418a98a3c203add0d4d4 already in progress: creating tablet
I20250112 20:51:38.812062 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]) successfully deleted
W20250112 20:51:38.812311 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 725e56fa8ac343e5b1ec1214da02b7d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725e56fa8ac343e5b1ec1214da02b7d4 already in progress: creating tablet
I20250112 20:51:38.812716 1163 ts_tablet_manager.cc:1428] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20250112 20:51:38.811628 1164 consensus_queue.cc:260] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.813194 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3b532068d2374b59a3c46525db7c50db (table default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]) successfully deleted
I20250112 20:51:38.814388 1163 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:38.815950 1164 ts_tablet_manager.cc:1428] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250112 20:51:38.817849 1164 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.819813 1048 raft_consensus.cc:3054] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.823159 2197 consensus_queue.cc:1035] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.824905 1178 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.825048 1178 ts_tablet_manager.cc:1397] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250112 20:51:38.825860 1178 raft_consensus.cc:357] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.826078 1178 raft_consensus.cc:383] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.826139 1178 raft_consensus.cc:738] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.826359 1178 consensus_queue.cc:260] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.826879 1178 ts_tablet_manager.cc:1428] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:38.827133 1178 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.828845 1178 tablet_bootstrap.cc:654] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.829655 1164 tablet_bootstrap.cc:654] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.839342 1163 tablet_bootstrap.cc:654] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.840674 1178 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.840818 1178 ts_tablet_manager.cc:1397] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250112 20:51:38.841658 1178 raft_consensus.cc:357] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.841842 1178 raft_consensus.cc:383] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.841902 1178 raft_consensus.cc:738] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.842103 1178 consensus_queue.cc:260] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.842612 1178 ts_tablet_manager.cc:1428] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:38.842856 1178 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.844673 1178 tablet_bootstrap.cc:654] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:38.845355 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 903027fe49e34a308108356ba89a9c3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903027fe49e34a308108356ba89a9c3a already in progress: creating tablet
I20250112 20:51:38.846560 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.846654 1164 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.846997 1164 ts_tablet_manager.cc:1397] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20250112 20:51:38.847133 880 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:38.848940 1164 raft_consensus.cc:357] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.849501 1164 raft_consensus.cc:383] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.849684 1164 raft_consensus.cc:738] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.850159 1186 tablet_replica.cc:331] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:38.850378 1186 raft_consensus.cc:2238] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.850594 1186 raft_consensus.cc:2267] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.850257 1164 consensus_queue.cc:260] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.851215 1186 ts_tablet_manager.cc:1905] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.853026 1164 ts_tablet_manager.cc:1428] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250112 20:51:38.851392 821 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:38.854017 1164 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.858345 1178 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.859091 1178 ts_tablet_manager.cc:1397] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250112 20:51:38.847391 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ca4d49496e42ddb263264ceef2f27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:38.859737 1164 tablet_bootstrap.cc:654] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.859951 1184 tablet_replica.cc:331] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:38.860165 1184 raft_consensus.cc:2238] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:38.860316 1184 raft_consensus.cc:2267] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:38.859894 1178 raft_consensus.cc:357] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.861199 1184 ts_tablet_manager.cc:1905] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:38.861231 1178 raft_consensus.cc:383] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.861296 1178 raft_consensus.cc:738] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.861476 1178 consensus_queue.cc:260] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.861933 1178 ts_tablet_manager.cc:1428] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:38.867259 1184 ts_tablet_manager.cc:1918] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:38.867411 1184 log.cc:1198] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/a3ca4d49496e42ddb263264ceef2f27a
I20250112 20:51:38.867769 1184 ts_tablet_manager.cc:1939] T a3ca4d49496e42ddb263264ceef2f27a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:38.868938 1178 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.870934 1178 tablet_bootstrap.cc:654] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:38.874222 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 903027fe49e34a308108356ba89a9c3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903027fe49e34a308108356ba89a9c3a already in progress: creating tablet
I20250112 20:51:38.899555 1163 tablet_bootstrap.cc:492] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:38.908843 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet a3ca4d49496e42ddb263264ceef2f27a (table default.Table_f594de5fea864b98a7b2838bf0257f3c [id=ad7dad3540d1406495302fef5a248367]) successfully deleted
I20250112 20:51:38.909601 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4303e4aec9694c49aca7c1532eaab3ed (table default.Table_90f38fee7fd8445a98f6af6d299318bc [id=f7bbf00f03b546d5bbcb872d503aaf4f]) successfully deleted
W20250112 20:51:38.909924 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3994d41a7ecd418a98a3c203add0d4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3994d41a7ecd418a98a3c203add0d4d4 already in progress: creating tablet
W20250112 20:51:38.910173 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 725e56fa8ac343e5b1ec1214da02b7d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725e56fa8ac343e5b1ec1214da02b7d4 already in progress: creating tablet
W20250112 20:51:38.910436 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 903027fe49e34a308108356ba89a9c3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 903027fe49e34a308108356ba89a9c3a already in progress: creating tablet
W20250112 20:51:38.910681 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d66c55c29de7458bb8ea69b826c567d7 because the tablet was not found. No further retry: Not found: Tablet not found: d66c55c29de7458bb8ea69b826c567d7
I20250112 20:51:38.912854 1178 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:38.914206 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8074b00eda8d434faaff4b98ae923ebe (DEFAULT_TABLE table=default.Table_cc39ddff5b22412fb084d77879afe4da [id=7fe9c7e16f9c46d3b1f163aa39f81b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.914613 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8074b00eda8d434faaff4b98ae923ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.921522 1164 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.921669 1164 ts_tablet_manager.cc:1397] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.001s
I20250112 20:51:38.922372 1164 raft_consensus.cc:357] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.922564 1164 raft_consensus.cc:383] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.922627 1164 raft_consensus.cc:738] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.922842 1164 consensus_queue.cc:260] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:38.923322 1164 ts_tablet_manager.cc:1428] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:38.923295 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11f04466bde745949068ba1eac6f68a4" } modify_external_catalogs: true
I20250112 20:51:38.923504 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11f04466bde745949068ba1eac6f68a4" } modify_external_catalogs: true
I20250112 20:51:38.923569 1164 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.926112 452 catalog_manager.cc:5526] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:38.926376 1164 tablet_bootstrap.cc:654] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.926539 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a218ef5825424b1b84c55b38b9b9023d (table default.Table_df950a6830fa45b1a8f21feda53ea0ea [id=05211d8527144710860dc2aca1ac76a3]). Expected version 1 got 0
I20250112 20:51:38.934728 1178 ts_tablet_manager.cc:1397] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.003s
I20250112 20:51:38.935616 1178 raft_consensus.cc:357] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.935835 1178 raft_consensus.cc:383] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.935900 1178 raft_consensus.cc:738] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.936085 1178 consensus_queue.cc:260] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.936628 1178 ts_tablet_manager.cc:1428] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:38.936923 1178 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:38.937433 2197 consensus_queue.cc:1035] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:38.940093 738 tablet_service.cc:1467] Processing CreateTablet for tablet 8074b00eda8d434faaff4b98ae923ebe (DEFAULT_TABLE table=default.Table_cc39ddff5b22412fb084d77879afe4da [id=7fe9c7e16f9c46d3b1f163aa39f81b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.940318 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8074b00eda8d434faaff4b98ae923ebe (DEFAULT_TABLE table=default.Table_cc39ddff5b22412fb084d77879afe4da [id=7fe9c7e16f9c46d3b1f163aa39f81b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.940487 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8074b00eda8d434faaff4b98ae923ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.940639 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8074b00eda8d434faaff4b98ae923ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.943863 737 tablet_service.cc:1467] Processing CreateTablet for tablet f9055531da4c453fa722eb7c78fa0e78 (DEFAULT_TABLE table=default.Table_fec3d1a320fe450dbab6cf50e1380d8d [id=1cd818b475f048178dc0eff23b8401e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.944764 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9055531da4c453fa722eb7c78fa0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.945019 1164 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.945232 1164 ts_tablet_manager.cc:1397] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
W20250112 20:51:38.945900 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8098069639a2435ab500908db9ecd12a (table default.Table_3886e5ca0a0b4c71ac1ade9fcf9d8ab2 [id=554666ac0524458dbb36e43d52dfbdaa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8098069639a2435ab500908db9ecd12a consensus state has no leader
I20250112 20:51:38.946398 1164 raft_consensus.cc:357] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.946723 1164 raft_consensus.cc:383] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.946861 1164 raft_consensus.cc:738] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.948413 1164 consensus_queue.cc:260] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.949595 1164 ts_tablet_manager.cc:1428] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250112 20:51:38.950052 1164 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.951557 2276 raft_consensus.cc:491] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.951711 2276 raft_consensus.cc:513] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.952487 2276 leader_election.cc:290] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.953028 1164 tablet_bootstrap.cc:654] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:38.953341 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a04c223889f424e831e82a52a84f804" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.953542 912 raft_consensus.cc:2463] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:38.953836 2276 raft_consensus.cc:491] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:38.953974 2276 raft_consensus.cc:513] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.954515 2276 leader_election.cc:290] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.955217 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f9055531da4c453fa722eb7c78fa0e78 (DEFAULT_TABLE table=default.Table_fec3d1a320fe450dbab6cf50e1380d8d [id=1cd818b475f048178dc0eff23b8401e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.955706 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9055531da4c453fa722eb7c78fa0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.954509 710 leader_election.cc:304] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.957804 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:38.957988 912 raft_consensus.cc:2463] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:38.958351 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:38.958415 710 leader_election.cc:304] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:38.958705 2208 raft_consensus.cc:2798] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.958801 2208 raft_consensus.cc:491] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.958910 2208 raft_consensus.cc:3054] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.960007 737 tablet_service.cc:1467] Processing CreateTablet for tablet 18664fc1f93f4d48a75223c80da29b7a (DEFAULT_TABLE table=default.Table_a9962f17769a4deaab73b90205bb15bd [id=e0a91f5a9b9f4a88a6e142778fba7550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.960176 1164 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:38.960294 885 tablet_service.cc:1467] Processing CreateTablet for tablet f9055531da4c453fa722eb7c78fa0e78 (DEFAULT_TABLE table=default.Table_fec3d1a320fe450dbab6cf50e1380d8d [id=1cd818b475f048178dc0eff23b8401e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.960376 1164 ts_tablet_manager.cc:1397] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20250112 20:51:38.960433 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18664fc1f93f4d48a75223c80da29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.960639 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9055531da4c453fa722eb7c78fa0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.961436 2208 raft_consensus.cc:513] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.961732 1164 raft_consensus.cc:357] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.962035 2208 leader_election.cc:290] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.962069 1164 raft_consensus.cc:383] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:38.962211 2208 raft_consensus.cc:2798] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:38.962191 1164 raft_consensus.cc:738] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:38.962291 2208 raft_consensus.cc:491] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:38.962373 2208 raft_consensus.cc:3054] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.962555 1164 consensus_queue.cc:260] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:38.963303 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dc719cd84124082b782016011e37c8b consensus state has no leader
I20250112 20:51:38.963394 1164 ts_tablet_manager.cc:1428] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:38.963838 1164 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:38.963966 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 3dba3d9315db4b9292288ee62c2244bc from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:38.965368 879 tablet_service.cc:1467] Processing CreateTablet for tablet 18664fc1f93f4d48a75223c80da29b7a (DEFAULT_TABLE table=default.Table_a9962f17769a4deaab73b90205bb15bd [id=e0a91f5a9b9f4a88a6e142778fba7550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.964891 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 18664fc1f93f4d48a75223c80da29b7a (DEFAULT_TABLE table=default.Table_a9962f17769a4deaab73b90205bb15bd [id=e0a91f5a9b9f4a88a6e142778fba7550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.965728 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18664fc1f93f4d48a75223c80da29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.965791 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18664fc1f93f4d48a75223c80da29b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.966204 889 tablet_service.cc:1467] Processing CreateTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 (DEFAULT_TABLE table=default.Table_11f04466bde745949068ba1eac6f68a4 [id=602b3e37fbc546da83f19b10975f6ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.966534 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e25ebef6384b14bf79cfb7af3a79b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.966831 754 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe (DEFAULT_TABLE table=default.Table_89dea959eeb74e07bf12c140416c0f75 [id=60932bffadda4f7599c023e0456c8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.967195 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfa15d0fd54baca3f4e1d91de676fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.967465 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa4249c43bb41fe8dec95e0c910a2c5 (DEFAULT_TABLE table=default.Table_acf981f928ef4a12a31795dec2c5256e [id=84cf2a69325041d68c596dd9bfe73fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.967804 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa4249c43bb41fe8dec95e0c910a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.968987 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa4249c43bb41fe8dec95e0c910a2c5 (DEFAULT_TABLE table=default.Table_acf981f928ef4a12a31795dec2c5256e [id=84cf2a69325041d68c596dd9bfe73fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.969195 874 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe (DEFAULT_TABLE table=default.Table_89dea959eeb74e07bf12c140416c0f75 [id=60932bffadda4f7599c023e0456c8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.969314 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa4249c43bb41fe8dec95e0c910a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.969480 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfa15d0fd54baca3f4e1d91de676fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.968987 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe (DEFAULT_TABLE table=default.Table_89dea959eeb74e07bf12c140416c0f75 [id=60932bffadda4f7599c023e0456c8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.970448 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfa15d0fd54baca3f4e1d91de676fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.973032 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 (DEFAULT_TABLE table=default.Table_11f04466bde745949068ba1eac6f68a4 [id=602b3e37fbc546da83f19b10975f6ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.973330 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e25ebef6384b14bf79cfb7af3a79b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.973346 1164 tablet_bootstrap.cc:654] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:38.974417 710 leader_election.cc:343] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.975119 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a04c223889f424e831e82a52a84f804" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:38.975360 912 raft_consensus.cc:3054] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:38.977349 2208 raft_consensus.cc:513] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:38.977356 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa4249c43bb41fe8dec95e0c910a2c5 (DEFAULT_TABLE table=default.Table_acf981f928ef4a12a31795dec2c5256e [id=84cf2a69325041d68c596dd9bfe73fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.977687 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa4249c43bb41fe8dec95e0c910a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.977926 747 tablet_service.cc:1467] Processing CreateTablet for tablet 940756f8867b4cde91fb0e89e6069265 (DEFAULT_TABLE table=default.Table_acc683b25e9a4c3f84ce1b9476963382 [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.978232 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940756f8867b4cde91fb0e89e6069265. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.978927 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 940756f8867b4cde91fb0e89e6069265 (DEFAULT_TABLE table=default.Table_acc683b25e9a4c3f84ce1b9476963382 [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.978931 738 tablet_service.cc:1467] Processing CreateTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 (DEFAULT_TABLE table=default.Table_11f04466bde745949068ba1eac6f68a4 [id=602b3e37fbc546da83f19b10975f6ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.979220 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940756f8867b4cde91fb0e89e6069265. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.979250 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e25ebef6384b14bf79cfb7af3a79b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.980515 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:38.980985 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a04c223889f424e831e82a52a84f804" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:38.981928 710 leader_election.cc:343] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:38.982132 710 leader_election.cc:343] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:38.984992 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a (DEFAULT_TABLE table=default.Table_5d48f415092e41a3ada61770f9908c9d [id=c685b81d7b7a479a83f287bd7580ce50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.985123 890 tablet_service.cc:1467] Processing CreateTablet for tablet 940756f8867b4cde91fb0e89e6069265 (DEFAULT_TABLE table=default.Table_acc683b25e9a4c3f84ce1b9476963382 [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.985318 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1dd19c189f4b1aa0faab3dac476a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.985421 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940756f8867b4cde91fb0e89e6069265. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.985455 2208 leader_election.cc:290] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:38.986958 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b519b0dc51a240629b6376767fc19e5d (DEFAULT_TABLE table=default.Table_06293bb20aca4d9696355a7fc3fdbe54 [id=bf209aeb3d874f0bbad07f03a42eedf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.987303 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b519b0dc51a240629b6376767fc19e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.988559 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a (DEFAULT_TABLE table=default.Table_5d48f415092e41a3ada61770f9908c9d [id=c685b81d7b7a479a83f287bd7580ce50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.988901 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1dd19c189f4b1aa0faab3dac476a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.989463 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a (DEFAULT_TABLE table=default.Table_5d48f415092e41a3ada61770f9908c9d [id=c685b81d7b7a479a83f287bd7580ce50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.989944 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1dd19c189f4b1aa0faab3dac476a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.993480 889 tablet_service.cc:1467] Processing CreateTablet for tablet b519b0dc51a240629b6376767fc19e5d (DEFAULT_TABLE table=default.Table_06293bb20aca4d9696355a7fc3fdbe54 [id=bf209aeb3d874f0bbad07f03a42eedf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.993685 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889 (DEFAULT_TABLE table=default.Table_2378bcfbef514d9a8e2bc42f74afada5 [id=2c990b9e60284083b5b8791167deaf29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.993897 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889 (DEFAULT_TABLE table=default.Table_2378bcfbef514d9a8e2bc42f74afada5 [id=2c990b9e60284083b5b8791167deaf29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.994068 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.994218 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.994415 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 998a2a37e8bc449aac8a064c3daae32a (DEFAULT_TABLE table=default.Table_9494be2cad4b41559c1d1563a64ce78f [id=78235b769ec24262985521988a094fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.994721 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998a2a37e8bc449aac8a064c3daae32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.995031 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889 (DEFAULT_TABLE table=default.Table_2378bcfbef514d9a8e2bc42f74afada5 [id=2c990b9e60284083b5b8791167deaf29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.995350 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d08ab7b6d4d49ecae42df7d6f8f4889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.995924 753 tablet_service.cc:1467] Processing CreateTablet for tablet 998a2a37e8bc449aac8a064c3daae32a (DEFAULT_TABLE table=default.Table_9494be2cad4b41559c1d1563a64ce78f [id=78235b769ec24262985521988a094fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.996243 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998a2a37e8bc449aac8a064c3daae32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.996380 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fae080f8e4b7436b9166c85604a77dfa (DEFAULT_TABLE table=default.Table_93a664f616254b98a1914cedad79b639 [id=e1281438874d4cc6ac4bd188844dd8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.996646 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fae080f8e4b7436b9166c85604a77dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.997057 893 tablet_service.cc:1467] Processing CreateTablet for tablet 998a2a37e8bc449aac8a064c3daae32a (DEFAULT_TABLE table=default.Table_9494be2cad4b41559c1d1563a64ce78f [id=78235b769ec24262985521988a094fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.998350 752 tablet_service.cc:1467] Processing CreateTablet for tablet 15b009608f514353be8a58408b4abdea (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.998684 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b009608f514353be8a58408b4abdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:38.999420 745 tablet_service.cc:1467] Processing CreateTablet for tablet fae080f8e4b7436b9166c85604a77dfa (DEFAULT_TABLE table=default.Table_93a664f616254b98a1914cedad79b639 [id=e1281438874d4cc6ac4bd188844dd8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:38.999720 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fae080f8e4b7436b9166c85604a77dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.001037 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 15b009608f514353be8a58408b4abdea (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.001030 886 tablet_service.cc:1467] Processing CreateTablet for tablet fae080f8e4b7436b9166c85604a77dfa (DEFAULT_TABLE table=default.Table_93a664f616254b98a1914cedad79b639 [id=e1281438874d4cc6ac4bd188844dd8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.001343 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fae080f8e4b7436b9166c85604a77dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.001343 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b009608f514353be8a58408b4abdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.006258 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.006433 910 raft_consensus.cc:3054] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.007876 912 raft_consensus.cc:2463] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.008535 747 tablet_service.cc:1467] Processing CreateTablet for tablet b519b0dc51a240629b6376767fc19e5d (DEFAULT_TABLE table=default.Table_06293bb20aca4d9696355a7fc3fdbe54 [id=bf209aeb3d874f0bbad07f03a42eedf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.008932 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b519b0dc51a240629b6376767fc19e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.009697 710 leader_election.cc:304] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.015882 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c37a19db737147b3a05d4cd80380f59e" } new_table_name: "default.Table_e5949130bc9c488da66e7ebeeaf6cc8c" modify_external_catalogs: true
I20250112 20:51:39.019645 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a04c223889f424e831e82a52a84f804" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.008636 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b519b0dc51a240629b6376767fc19e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.026001 910 raft_consensus.cc:2463] T e912d588104f482498c40239c9e585b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.027957 878 tablet_service.cc:1467] Processing CreateTablet for tablet 15b009608f514353be8a58408b4abdea (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.028338 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b009608f514353be8a58408b4abdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.046540 1178 tablet_bootstrap.cc:654] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.055613 2208 raft_consensus.cc:491] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.055951 1163 ts_tablet_manager.cc:1397] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.242s user 0.008s sys 0.013s
I20250112 20:51:39.055878 2208 raft_consensus.cc:513] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.056015 1186 ts_tablet_manager.cc:1918] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.056555 2208 leader_election.cc:290] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.056577 1163 raft_consensus.cc:357] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.056604 1186 log.cc:1198] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:39.058674 1186 ts_tablet_manager.cc:1939] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:39.060685 1163 raft_consensus.cc:383] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.062461 1178 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.062592 1163 raft_consensus.cc:738] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.062749 1178 ts_tablet_manager.cc:1397] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.126s user 0.007s sys 0.000s
I20250112 20:51:39.062940 1163 consensus_queue.cc:260] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.063452 1163 ts_tablet_manager.cc:1428] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:39.063985 1163 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.065088 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
I20250112 20:51:39.065407 1163 tablet_bootstrap.cc:654] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.066248 1186 tablet_replica.cc:331] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:39.065605 1178 raft_consensus.cc:357] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.066738 1178 raft_consensus.cc:383] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.067086 1178 raft_consensus.cc:738] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.067381 1186 raft_consensus.cc:2238] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.068125 1186 raft_consensus.cc:2267] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.068262 1178 consensus_queue.cc:260] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.070959 1186 ts_tablet_manager.cc:1905] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.071249 1178 ts_tablet_manager.cc:1428] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 20:51:39.072540 1178 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.076666 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998a2a37e8bc449aac8a064c3daae32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.084947 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1a9c6d59d5be4e2f9c46487c4f674ad7
I20250112 20:51:39.085412 454 catalog_manager.cc:6596] ReplaceTablet: tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 deleted and replaced by tablet 72252e88e6664bab850663236d280987
I20250112 20:51:39.085628 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 72252e88e6664bab850663236d280987) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.085680 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e6b4882a3d744bda5677aa9e01a53f4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:39.086884 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 08cfb20cbf384b559bea4528e5f37bc2
I20250112 20:51:39.088534 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 868ce9397b884b0095dae39ea57f5fd9
I20250112 20:51:39.089283 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.089725 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.090111 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d9ade1d7332e45dbaffe3d2d2382f568
I20250112 20:51:39.090687 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.091866 737 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.091850 1178 tablet_bootstrap.cc:654] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.092448 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:39.093578 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.093664 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d384628ba9b4f81b768eda4f796632b from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:39.094146 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_56d64a03c356468599c3310df24db01b" } new_table_name: "default.Table_424524dba512432681b74ca38ea110ff" modify_external_catalogs: true
I20250112 20:51:39.096491 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aeb009faaf764c289ecacebf52f2fcdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.097061 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 72252e88e6664bab850663236d280987) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.097158 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb009faaf764c289ecacebf52f2fcdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.097265 1185 tablet_replica.cc:331] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:39.097368 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.097441 1185 raft_consensus.cc:2238] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.097613 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.097769 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ade1d7332e45dbaffe3d2d2382f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.098526 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7f122343e2d74bf7b3eefea9de3ff725" } modify_external_catalogs: true
I20250112 20:51:39.098675 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7f122343e2d74bf7b3eefea9de3ff725" } modify_external_catalogs: true
I20250112 20:51:39.100001 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e25daa065bc34cef93655a371805ccf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.100459 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e25daa065bc34cef93655a371805ccf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.103619 1184 tablet_replica.cc:331] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.104030 1184 raft_consensus.cc:2238] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.104179 1184 raft_consensus.cc:2267] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.104404 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08048c402345a686d1e3fb3039bbc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.104710 1184 ts_tablet_manager.cc:1905] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.105037 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 15e25ebef6384b14bf79cfb7af3a79b6
I20250112 20:51:39.105628 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e9dd5f9cbc524e449863f6c9b52424b6" } new_table_name: "default.Table_2767c4e6cdbe4f798cc72acefaa386f7" modify_external_catalogs: true
W20250112 20:51:39.107571 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d9ade1d7332e45dbaffe3d2d2382f568 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ade1d7332e45dbaffe3d2d2382f568 already in progress: creating tablet
W20250112 20:51:39.107846 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 868ce9397b884b0095dae39ea57f5fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 868ce9397b884b0095dae39ea57f5fd9 already in progress: creating tablet
I20250112 20:51:39.108309 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.108822 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.110441 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 15e25ebef6384b14bf79cfb7af3a79b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.112337 452 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:39.113756 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_861997ae234b444fae5e7160ed43a3e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.113991 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d8e68746cf444ebac5e22728ab6f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.114223 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_861997ae234b444fae5e7160ed43a3e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:39.117334 847 consensus_peers.cc:487] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:39.117465 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 72252e88e6664bab850663236d280987) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.124606 1184 ts_tablet_manager.cc:1918] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:39.124821 1184 log.cc:1198] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1a9c6d59d5be4e2f9c46487c4f674ad7
I20250112 20:51:39.125237 1184 ts_tablet_manager.cc:1939] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:39.126338 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (table default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]) successfully deleted
W20250112 20:51:39.127022 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 08cfb20cbf384b559bea4528e5f37bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cfb20cbf384b559bea4528e5f37bc2 already in progress: creating tablet
W20250112 20:51:39.127617 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 868ce9397b884b0095dae39ea57f5fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 868ce9397b884b0095dae39ea57f5fd9 already in progress: creating tablet
W20250112 20:51:39.128260 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d9ade1d7332e45dbaffe3d2d2382f568 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ade1d7332e45dbaffe3d2d2382f568 already in progress: creating tablet
I20250112 20:51:39.128319 1163 tablet_bootstrap.cc:492] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.128491 1163 ts_tablet_manager.cc:1397] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.002s
I20250112 20:51:39.128547 1184 tablet_replica.cc:331] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.128751 1184 raft_consensus.cc:2238] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.128947 1184 raft_consensus.cc:2267] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.129314 1163 raft_consensus.cc:357] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.129602 1163 raft_consensus.cc:383] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.129647 1184 ts_tablet_manager.cc:1905] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.129696 1163 raft_consensus.cc:738] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.130028 1163 consensus_queue.cc:260] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.130565 1163 ts_tablet_manager.cc:1428] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:39.133224 1163 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.135903 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3dba3d9315db4b9292288ee62c2244bc
I20250112 20:51:39.136377 451 catalog_manager.cc:6596] ReplaceTablet: tablet 3dba3d9315db4b9292288ee62c2244bc deleted and replaced by tablet 3846863c74f64923bda6e5a19e7f2770
I20250112 20:51:39.137123 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:39.137468 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ccb593413124921af94d0b855789b52 consensus state has no leader
I20250112 20:51:39.137504 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.138435 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.138851 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f6ad2ac50c745448031bdf94e0eea59 consensus state has no leader
I20250112 20:51:39.139072 1163 tablet_bootstrap.cc:654] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.140022 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 24a5162a25ce47519a091742f3d5ca31
I20250112 20:51:39.140816 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.141012 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.142199 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:39.142201 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.142719 454 catalog_manager.cc:6596] ReplaceTablet: tablet 2e6b4882a3d744bda5677aa9e01a53f4 deleted and replaced by tablet 6921185b545244978f8fdeafd1fb6c5e
W20250112 20:51:39.144059 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 902fc636a0494462ae1e73b0cefc5dfe consensus state has no leader
I20250112 20:51:39.144328 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b85b9e82f9e847c4acfeecb82201e694 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:39.147112 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cfb20cbf384b559bea4528e5f37bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.148211 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.148619 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.148801 1163 tablet_bootstrap.cc:492] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.149132 1163 ts_tablet_manager.cc:1397] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20250112 20:51:39.150084 1163 raft_consensus.cc:357] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.150318 1163 raft_consensus.cc:383] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.150408 1163 raft_consensus.cc:738] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.150655 1163 consensus_queue.cc:260] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.151176 1163 ts_tablet_manager.cc:1428] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:39.151646 1163 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.151882 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1b79e8a8774d47a6a96ca46ce2427b18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.152187 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
W20250112 20:51:39.152562 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b79e8a8774d47a6a96ca46ce2427b18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.154623 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d60604713524dba8f3a42c4f3bc7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.154793 1186 ts_tablet_manager.cc:1918] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.155615 1163 tablet_bootstrap.cc:654] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.155593 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6f7b30f2842340de9879b5e275a9c8a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.155977 1186 log.cc:1198] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b310e4135c2b424fb006a38a60e41066
W20250112 20:51:39.156241 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f7b30f2842340de9879b5e275a9c8a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.159016 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6b2acbe71aac4af0978c56cb845b5dee" } new_table_name: "default.Table_da12b2e64bd640d9b17b825b8653b314" modify_external_catalogs: true
I20250112 20:51:39.160529 1184 ts_tablet_manager.cc:1918] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.160627 1184 log.cc:1198] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8d08048c402345a686d1e3fb3039bbc1
I20250112 20:51:39.160979 1184 ts_tablet_manager.cc:1939] T 8d08048c402345a686d1e3fb3039bbc1 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:39.161489 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_90bbde70dc014c87be25b842e16d44d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.162037 1186 ts_tablet_manager.cc:1939] T b310e4135c2b424fb006a38a60e41066 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:39.162016 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8d08048c402345a686d1e3fb3039bbc1 (table default.Table_3c1abf71e9154b45ba412645d3f22386 [id=846e346a1c2f4d9baf54c8f4ac5256b6]) successfully deleted
W20250112 20:51:39.162110 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90bbde70dc014c87be25b842e16d44d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:39.162662 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 15e25ebef6384b14bf79cfb7af3a79b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e25ebef6384b14bf79cfb7af3a79b6 already in progress: creating tablet
I20250112 20:51:39.163367 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_19cd3e26ebaa41008374cb9290c3e323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.163779 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19cd3e26ebaa41008374cb9290c3e323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.164064 1184 tablet_replica.cc:331] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.164248 1184 raft_consensus.cc:2238] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.164376 1184 raft_consensus.cc:2267] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.165459 1184 ts_tablet_manager.cc:1905] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.166199 1163 tablet_bootstrap.cc:492] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.166339 1163 ts_tablet_manager.cc:1397] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20250112 20:51:39.166417 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b310e4135c2b424fb006a38a60e41066 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
I20250112 20:51:39.167048 1163 raft_consensus.cc:357] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.167306 1163 raft_consensus.cc:383] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.167368 1163 raft_consensus.cc:738] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.167590 1163 consensus_queue.cc:260] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.168150 1163 ts_tablet_manager.cc:1428] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:39.168519 1163 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.170058 1163 tablet_bootstrap.cc:654] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:39.171295 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 08cfb20cbf384b559bea4528e5f37bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cfb20cbf384b559bea4528e5f37bc2 already in progress: creating tablet
I20250112 20:51:39.148053 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.171789 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 868ce9397b884b0095dae39ea57f5fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 868ce9397b884b0095dae39ea57f5fd9 already in progress: creating tablet
W20250112 20:51:39.172044 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d9ade1d7332e45dbaffe3d2d2382f568 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ade1d7332e45dbaffe3d2d2382f568 already in progress: creating tablet
I20250112 20:51:39.172690 1184 ts_tablet_manager.cc:1918] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.172816 1184 log.cc:1198] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/95d8e68746cf444ebac5e22728ab6f34
I20250112 20:51:39.173131 1184 ts_tablet_manager.cc:1939] T 95d8e68746cf444ebac5e22728ab6f34 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:39.173544 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 95d8e68746cf444ebac5e22728ab6f34 (table default.Table_43f6a278ed0b4388aceea8a274566af9 [id=2b95ff89bfc5461d9b317edf6ac3ec88]) successfully deleted
W20250112 20:51:39.174188 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3dba3d9315db4b9292288ee62c2244bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dba3d9315db4b9292288ee62c2244bc already in progress: creating tablet
I20250112 20:51:39.176986 1186 tablet_replica.cc:331] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:39.177119 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 15e25ebef6384b14bf79cfb7af3a79b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e25ebef6384b14bf79cfb7af3a79b6 already in progress: creating tablet
W20250112 20:51:39.177922 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 24a5162a25ce47519a091742f3d5ca31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a5162a25ce47519a091742f3d5ca31 already in progress: creating tablet
I20250112 20:51:39.178193 1186 raft_consensus.cc:2238] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:39.178562 1186 raft_consensus.cc:2267] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.178779 1184 tablet_replica.cc:331] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.178949 1184 raft_consensus.cc:2238] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.179096 1184 raft_consensus.cc:2267] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:39.179121 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2e6b4882a3d744bda5677aa9e01a53f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6b4882a3d744bda5677aa9e01a53f4 already in progress: creating tablet
I20250112 20:51:39.180074 1163 tablet_bootstrap.cc:492] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.180215 1163 ts_tablet_manager.cc:1397] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250112 20:51:39.180418 1186 ts_tablet_manager.cc:1905] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.180859 1163 raft_consensus.cc:357] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.181068 1163 raft_consensus.cc:383] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.181162 1163 raft_consensus.cc:738] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.181416 1163 consensus_queue.cc:260] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.181890 1163 ts_tablet_manager.cc:1428] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:39.181922 1184 ts_tablet_manager.cc:1905] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.182168 1163 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.183405 1163 tablet_bootstrap.cc:654] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.192360 2208 raft_consensus.cc:2798] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.192456 1185 raft_consensus.cc:2267] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.192658 2276 raft_consensus.cc:491] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.192813 2276 raft_consensus.cc:513] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.193135 1185 ts_tablet_manager.cc:1905] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.193569 2276 leader_election.cc:290] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.194016 2276 raft_consensus.cc:491] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.194133 2276 raft_consensus.cc:513] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.194806 2276 leader_election.cc:290] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.195297 2276 raft_consensus.cc:491] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.195415 2276 raft_consensus.cc:513] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.196029 2276 leader_election.cc:290] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.200531 1185 ts_tablet_manager.cc:1918] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:39.200693 1185 log.cc:1198] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1a9c6d59d5be4e2f9c46487c4f674ad7
W20250112 20:51:39.201140 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 191467us
I20250112 20:51:39.201143 1185 ts_tablet_manager.cc:1939] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:39.203387 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_22f40a2d9f084970b9d2152f401edd49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.204123 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22f40a2d9f084970b9d2152f401edd49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.207898 2208 raft_consensus.cc:695] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.208173 2208 consensus_queue.cc:237] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:39.210491 710 leader_election.cc:343] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.210443 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f79b3309e45afb9e18cad97ddfeb0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.210834 710 leader_election.cc:304] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.210906 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.210947 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.210950 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.211508 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.211756 2325 raft_consensus.cc:491] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.211850 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.211889 2325 raft_consensus.cc:513] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.212023 910 raft_consensus.cc:2463] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.212491 2325 leader_election.cc:290] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.212628 2325 raft_consensus.cc:2798] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.212733 2325 raft_consensus.cc:695] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.213377 1186 ts_tablet_manager.cc:1918] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:39.213547 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.213842 1163 tablet_bootstrap.cc:492] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.213974 1163 ts_tablet_manager.cc:1397] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.003s
I20250112 20:51:39.214694 1163 raft_consensus.cc:357] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.214890 1163 raft_consensus.cc:383] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.214946 1163 raft_consensus.cc:738] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.215148 1163 consensus_queue.cc:260] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.215662 1163 ts_tablet_manager.cc:1428] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:39.215963 1163 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.216341 1186 log.cc:1198] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1a9c6d59d5be4e2f9c46487c4f674ad7
I20250112 20:51:39.216709 1186 ts_tablet_manager.cc:1939] T 1a9c6d59d5be4e2f9c46487c4f674ad7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:39.216850 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (table default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]) successfully deleted
I20250112 20:51:39.217167 2325 consensus_queue.cc:237] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:39.217265 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 15e25ebef6384b14bf79cfb7af3a79b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e25ebef6384b14bf79cfb7af3a79b6 already in progress: creating tablet
I20250112 20:51:39.217499 1178 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
W20250112 20:51:39.217618 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3dba3d9315db4b9292288ee62c2244bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dba3d9315db4b9292288ee62c2244bc already in progress: creating tablet
W20250112 20:51:39.217942 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 24a5162a25ce47519a091742f3d5ca31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a5162a25ce47519a091742f3d5ca31 already in progress: creating tablet
I20250112 20:51:39.217976 1178 ts_tablet_manager.cc:1397] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.013s
W20250112 20:51:39.218382 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2e6b4882a3d744bda5677aa9e01a53f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6b4882a3d744bda5677aa9e01a53f4 already in progress: creating tablet
W20250112 20:51:39.218691 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 08cfb20cbf384b559bea4528e5f37bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cfb20cbf384b559bea4528e5f37bc2 already in progress: creating tablet
I20250112 20:51:39.219264 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1a9c6d59d5be4e2f9c46487c4f674ad7 (table default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]) successfully deleted
W20250112 20:51:39.220057 710 leader_election.cc:343] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.220288 710 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.220300 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4301a07fcb044fd2a6f3e5ef424b411f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.220476 913 raft_consensus.cc:2463] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.220700 1178 raft_consensus.cc:357] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.220517 710 leader_election.cc:304] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.221498 2208 raft_consensus.cc:2798] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.221495 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4301a07fcb044fd2a6f3e5ef424b411f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.221334 1178 raft_consensus.cc:383] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.221606 2208 raft_consensus.cc:491] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.221673 2208 raft_consensus.cc:3054] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.221644 1178 raft_consensus.cc:738] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.222199 1178 consensus_queue.cc:260] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.223974 2208 raft_consensus.cc:513] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.224485 2208 leader_election.cc:290] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:39.224746 710 leader_election.cc:343] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:39.225039 710 leader_election.cc:304] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.225072 1178 ts_tablet_manager.cc:1428] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250112 20:51:39.225253 710 leader_election.cc:343] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.225329 710 leader_election.cc:304] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:39.225629 710 leader_election.cc:343] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.226037 710 leader_election.cc:343] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.226114 710 leader_election.cc:304] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:39.226867 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3dba3d9315db4b9292288ee62c2244bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dba3d9315db4b9292288ee62c2244bc already in progress: creating tablet
W20250112 20:51:39.227473 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 24a5162a25ce47519a091742f3d5ca31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a5162a25ce47519a091742f3d5ca31 already in progress: creating tablet
I20250112 20:51:39.227705 1178 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:39.228380 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2e6b4882a3d744bda5677aa9e01a53f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6b4882a3d744bda5677aa9e01a53f4 already in progress: creating tablet
I20250112 20:51:39.229970 1044 raft_consensus.cc:3054] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.230208 1164 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.230252 2059 raft_consensus.cc:2798] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.230314 1164 ts_tablet_manager.cc:1397] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.267s user 0.019s sys 0.017s
I20250112 20:51:39.230322 2059 raft_consensus.cc:491] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.230419 2059 raft_consensus.cc:3054] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.230556 2323 raft_consensus.cc:2743] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.231137 1164 raft_consensus.cc:357] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.231343 1164 raft_consensus.cc:383] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.231390 1164 raft_consensus.cc:738] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.231545 1164 consensus_queue.cc:260] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.232036 1164 ts_tablet_manager.cc:1428] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:39.232249 1164 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:39.232947 2059 raft_consensus.cc:513] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.233443 1164 tablet_bootstrap.cc:654] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.233455 2059 leader_election.cc:290] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.233630 2208 raft_consensus.cc:2743] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:39.233990 710 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.234077 710 leader_election.cc:304] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:39.234253 1163 tablet_bootstrap.cc:654] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:39.234404 710 leader_election.cc:343] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.234967 2208 raft_consensus.cc:2743] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.235015 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:39.235450 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4301a07fcb044fd2a6f3e5ef424b411f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:39.235793 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.236008 913 raft_consensus.cc:3054] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.238570 913 raft_consensus.cc:2463] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.238847 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4301a07fcb044fd2a6f3e5ef424b411f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.239022 912 raft_consensus.cc:3054] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.239084 1178 tablet_bootstrap.cc:654] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.243458 912 raft_consensus.cc:2463] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:39.244009 710 leader_election.cc:343] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.244222 710 leader_election.cc:343] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.244377 710 leader_election.cc:304] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.244601 2325 raft_consensus.cc:2798] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.244791 2325 raft_consensus.cc:695] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.244762 710 leader_election.cc:304] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.245064 2208 raft_consensus.cc:2798] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.245182 2208 raft_consensus.cc:695] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.245410 2208 consensus_queue.cc:237] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:39.245620 847 consensus_peers.cc:487] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:39.245663 2325 consensus_queue.cc:237] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.246999 774 raft_consensus.cc:1270] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:39.247231 2076 consensus_queue.cc:1035] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.247627 2076 consensus_queue.cc:1035] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.266991 1178 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.269896 1163 tablet_bootstrap.cc:492] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.268366 1178 ts_tablet_manager.cc:1397] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.003s
I20250112 20:51:39.270192 1163 ts_tablet_manager.cc:1397] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.003s
I20250112 20:51:39.270798 1164 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.272056 1164 ts_tablet_manager.cc:1397] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.001s
I20250112 20:51:39.275151 1045 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.275379 2296 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.275651 1163 raft_consensus.cc:357] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.275605 1178 raft_consensus.cc:357] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.276925 1163 raft_consensus.cc:383] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.277063 1178 raft_consensus.cc:383] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.277312 1163 raft_consensus.cc:738] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.277453 1178 raft_consensus.cc:738] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.277403 1164 raft_consensus.cc:357] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.278685 1164 raft_consensus.cc:383] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.279065 1164 raft_consensus.cc:738] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.278551 1163 consensus_queue.cc:260] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.278611 1178 consensus_queue.cc:260] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.280092 1164 consensus_queue.cc:260] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.281723 1178 ts_tablet_manager.cc:1428] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 20:51:39.281852 1163 ts_tablet_manager.cc:1428] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:39.282743 1164 ts_tablet_manager.cc:1428] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:39.282959 1163 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.283443 1178 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.284632 1164 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:39.291164 1178 tablet_bootstrap.cc:654] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.292104 1163 tablet_bootstrap.cc:654] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.292158 1164 tablet_bootstrap.cc:654] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:39.299753 847 consensus_peers.cc:487] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:39.302873 2140 raft_consensus.cc:491] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.303028 2140 raft_consensus.cc:513] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.303911 2140 leader_election.cc:290] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.304764 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.305035 912 raft_consensus.cc:2463] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:39.305397 983 leader_election.cc:304] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:39.305668 2140 raft_consensus.cc:2798] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:39.305738 2140 raft_consensus.cc:491] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.305811 2140 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.306447 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:39.307394 1982 consensus_queue.cc:1035] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.308053 2325 consensus_queue.cc:1035] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.308156 1982 consensus_queue.cc:1035] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.310521 2140 raft_consensus.cc:513] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.310987 1044 raft_consensus.cc:1235] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Rejecting Update request from peer aefe1db8b19244d891b6bcd8971e9b17 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250112 20:51:39.312501 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.313136 912 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.315646 912 raft_consensus.cc:2463] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 2.
I20250112 20:51:39.316684 983 leader_election.cc:304] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:39.317363 2208 consensus_queue.cc:1046] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.317639 2325 raft_consensus.cc:3049] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 20:51:39.317734 2325 raft_consensus.cc:738] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.318017 2325 consensus_queue.cc:260] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.318389 2325 raft_consensus.cc:3054] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.321636 2140 leader_election.cc:290] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.321830 2140 raft_consensus.cc:2798] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:39.321940 2140 raft_consensus.cc:695] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:39.322185 2140 consensus_queue.cc:237] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.328490 1184 ts_tablet_manager.cc:1918] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.330320 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06253bb13a9541d8b408d1bf73e469dd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:39.330657 770 raft_consensus.cc:2405] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.330826 2339 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.331640 1178 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.331871 1178 ts_tablet_manager.cc:1397] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20250112 20:51:39.334457 1184 log.cc:1198] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1d60604713524dba8f3a42c4f3bc7300
I20250112 20:51:39.335000 1184 ts_tablet_manager.cc:1939] T 1d60604713524dba8f3a42c4f3bc7300 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:39.335835 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1d60604713524dba8f3a42c4f3bc7300 (table default.Table_326496344cf44e9d949ee3545f916c40 [id=463a3bb9a3d9415e923e17d7951087f9]) successfully deleted
I20250112 20:51:39.338837 1163 tablet_bootstrap.cc:492] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.338783 1164 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.342832 1163 ts_tablet_manager.cc:1397] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s
I20250112 20:51:39.343017 1164 ts_tablet_manager.cc:1397] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.001s
I20250112 20:51:39.342290 1178 raft_consensus.cc:357] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.344233 1178 raft_consensus.cc:383] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.345145 1178 raft_consensus.cc:738] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:39.349092 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e92d58058511437d830a8f3cbba1e206 consensus state has no leader
I20250112 20:51:39.349648 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:39.349951 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:39.350160 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]). Expected version 1 got 0
I20250112 20:51:39.348501 1178 consensus_queue.cc:260] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.351639 2140 raft_consensus.cc:491] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.351950 2140 raft_consensus.cc:513] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.352540 2140 leader_election.cc:290] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.352818 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 38cdb9a415f745cbb62f27763e490d5b (table default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7]). Expected version 1 got 0
I20250112 20:51:39.353036 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet a218ef5825424b1b84c55b38b9b9023d (table default.Table_df950a6830fa45b1a8f21feda53ea0ea [id=05211d8527144710860dc2aca1ac76a3]). Expected version 1 got 0
I20250112 20:51:39.353142 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.353310 450 catalog_manager.cc:5526] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:39.351563 1163 raft_consensus.cc:357] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.351902 1164 raft_consensus.cc:357] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.353691 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:39.353685 1163 raft_consensus.cc:383] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.354135 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 55d75f9b410f4790ac926646a3cf0fd0 (table default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8]). Expected version 1 got 0
I20250112 20:51:39.353755 1164 raft_consensus.cc:383] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.354436 1163 raft_consensus.cc:738] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.354378 1164 raft_consensus.cc:738] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.355480 889 tablet_service.cc:1467] Processing CreateTablet for tablet f321ea974d97408a91f0833eb63dd415 (DEFAULT_TABLE table=default.Table_0e3604a7e91842a99e0046e67f3cb0cf [id=93d45a6665f7418f8865f68e050d74ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.355621 1178 ts_tablet_manager.cc:1428] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20250112 20:51:39.356470 1163 consensus_queue.cc:260] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.356345 1164 consensus_queue.cc:260] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.357818 1178 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.358424 2325 raft_consensus.cc:491] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.358534 2325 raft_consensus.cc:513] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.358574 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 (DEFAULT_TABLE table=default.Table_088aef221fd440b1a66db4f07efa4232 [id=2647fff11ff442428f38b05b2515c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.358916 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9202294cd5d4aa9b5396ebdd65abda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.359090 2325 leader_election.cc:290] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.360378 893 tablet_service.cc:1467] Processing CreateTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 (DEFAULT_TABLE table=default.Table_088aef221fd440b1a66db4f07efa4232 [id=2647fff11ff442428f38b05b2515c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.360479 1163 ts_tablet_manager.cc:1428] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250112 20:51:39.360543 1164 ts_tablet_manager.cc:1428] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20250112 20:51:39.361011 1010 tablet_service.cc:1467] Processing CreateTablet for tablet f321ea974d97408a91f0833eb63dd415 (DEFAULT_TABLE table=default.Table_0e3604a7e91842a99e0046e67f3cb0cf [id=93d45a6665f7418f8865f68e050d74ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.362030 1163 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.361968 1164 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:39.364414 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321ea974d97408a91f0833eb63dd415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.371992 1163 tablet_bootstrap.cc:654] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.374291 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:39.379680 1163 tablet_bootstrap.cc:492] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.379983 1163 ts_tablet_manager.cc:1397] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20250112 20:51:39.380378 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6995f7fbbf480995353b66749fbc10 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.380918 878 tablet_service.cc:1467] Processing CreateTablet for tablet 1748fb558c4d48f49432840055d71c67 (DEFAULT_TABLE table=default.Table_e236e2ca4b654f28a0175e3f7041d249 [id=58125b0502e54ae8a34f91861f15a171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.381906 747 tablet_service.cc:1467] Processing CreateTablet for tablet f321ea974d97408a91f0833eb63dd415 (DEFAULT_TABLE table=default.Table_0e3604a7e91842a99e0046e67f3cb0cf [id=93d45a6665f7418f8865f68e050d74ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.382272 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321ea974d97408a91f0833eb63dd415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.382359 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5aca1ff0fd541bfa96bf7f279fa4e2d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:39.381981 1163 raft_consensus.cc:357] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.383008 1163 raft_consensus.cc:383] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.383147 738 tablet_service.cc:1467] Processing CreateTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 (DEFAULT_TABLE table=default.Table_088aef221fd440b1a66db4f07efa4232 [id=2647fff11ff442428f38b05b2515c932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.383208 1163 raft_consensus.cc:738] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.383445 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9202294cd5d4aa9b5396ebdd65abda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.383721 1163 consensus_queue.cc:260] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.385242 1163 ts_tablet_manager.cc:1428] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:39.385247 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.385442 912 raft_consensus.cc:2463] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.385677 754 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6995f7fbbf480995353b66749fbc10 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.386013 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f6995f7fbbf480995353b66749fbc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.385919 1163 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.390108 1163 tablet_bootstrap.cc:654] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.391168 885 tablet_service.cc:1467] Processing CreateTablet for tablet 21d79137148f456386b1a6f49695c691 (DEFAULT_TABLE table=default.Table_073ac55e6a934b0b943cfb161a831bb3 [id=83c8c92c7e4a41729d173a4822048ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.391757 958 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:39.392683 983 leader_election.cc:304] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:39.393180 2140 raft_consensus.cc:2743] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.393507 2325 raft_consensus.cc:491] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.393637 2325 raft_consensus.cc:513] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.394311 2325 leader_election.cc:290] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.394750 710 leader_election.cc:304] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.395187 747 tablet_service.cc:1467] Processing CreateTablet for tablet 21d79137148f456386b1a6f49695c691 (DEFAULT_TABLE table=default.Table_073ac55e6a934b0b943cfb161a831bb3 [id=83c8c92c7e4a41729d173a4822048ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.399667 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d79137148f456386b1a6f49695c691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.399724 2325 raft_consensus.cc:2798] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.399833 2325 raft_consensus.cc:491] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.399916 2325 raft_consensus.cc:3054] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.402891 2325 raft_consensus.cc:513] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.403581 2325 leader_election.cc:290] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.404912 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.405133 913 raft_consensus.cc:3054] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.407330 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1748fb558c4d48f49432840055d71c67 (DEFAULT_TABLE table=default.Table_e236e2ca4b654f28a0175e3f7041d249 [id=58125b0502e54ae8a34f91861f15a171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.407616 913 raft_consensus.cc:2463] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.407759 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748fb558c4d48f49432840055d71c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.408341 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.408524 912 raft_consensus.cc:2463] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.416703 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9202294cd5d4aa9b5396ebdd65abda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.416766 2326 raft_consensus.cc:491] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.416962 2326 raft_consensus.cc:513] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.417513 2326 leader_election.cc:290] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.419710 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321ea974d97408a91f0833eb63dd415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.423161 1164 tablet_bootstrap.cc:654] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.423597 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d79137148f456386b1a6f49695c691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.427338 2197 raft_consensus.cc:491] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.427471 2197 raft_consensus.cc:513] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.428100 2197 leader_election.cc:290] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.428694 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748fb558c4d48f49432840055d71c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.430140 1178 tablet_bootstrap.cc:654] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.432687 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cc55237c53344dfbbf00a7b33b6962e9" } modify_external_catalogs: true
I20250112 20:51:39.432922 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cc55237c53344dfbbf00a7b33b6962e9" } modify_external_catalogs: true
I20250112 20:51:39.433266 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f6995f7fbbf480995353b66749fbc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.435343 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_53510fe9a4124c20880d08d01510903c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.435910 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53510fe9a4124c20880d08d01510903c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.437147 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9" } modify_external_catalogs: true
I20250112 20:51:39.437312 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9" } modify_external_catalogs: true
I20250112 20:51:39.437822 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c06477d484e0483589b0530cf5787b9c" } new_table_name: "default.Table_86a42a16a5274fd4a667d4b7df06b8ea" modify_external_catalogs: true
I20250112 20:51:39.439711 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_89dea959eeb74e07bf12c140416c0f75" } modify_external_catalogs: true
I20250112 20:51:39.439899 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_89dea959eeb74e07bf12c140416c0f75" } modify_external_catalogs: true
W20250112 20:51:39.441280 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:39.442840 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 21d79137148f456386b1a6f49695c691 (DEFAULT_TABLE table=default.Table_073ac55e6a934b0b943cfb161a831bb3 [id=83c8c92c7e4a41729d173a4822048ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.442935 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.443233 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d79137148f456386b1a6f49695c691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.443269 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:39.445319 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b4de1d4b39114ec2b50c35a8a02318f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.445811 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4de1d4b39114ec2b50c35a8a02318f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.450377 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.450951 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 1748fb558c4d48f49432840055d71c67 (DEFAULT_TABLE table=default.Table_e236e2ca4b654f28a0175e3f7041d249 [id=58125b0502e54ae8a34f91861f15a171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.451328 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748fb558c4d48f49432840055d71c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.457665 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8f6995f7fbbf480995353b66749fbc10 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.458118 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f6995f7fbbf480995353b66749fbc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.465377 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:39.465986 453 catalog_manager.cc:6596] ReplaceTablet: tablet b85b9e82f9e847c4acfeecb82201e694 deleted and replaced by tablet e0bbf979d0b841899a2990a39d83394e
I20250112 20:51:39.468987 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1a61b32d430d45e7875d19a9905ef7fc" } modify_external_catalogs: true
I20250112 20:51:39.469202 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1a61b32d430d45e7875d19a9905ef7fc" } modify_external_catalogs: true
I20250112 20:51:39.470283 879 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.472306 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.475041 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b85b9e82f9e847c4acfeecb82201e694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85b9e82f9e847c4acfeecb82201e694 already in progress: creating tablet
I20250112 20:51:39.478608 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7] alter complete (version 1)
I20250112 20:51:39.482419 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:39.485605 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 22f6b2c4ea5f4167896c78b1c64cbb62
I20250112 20:51:39.487540 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.488288 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6b2c4ea5f4167896c78b1c64cbb62 already in progress: creating tablet
W20250112 20:51:39.493088 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05550152671e4e28829b5b102b59f715 consensus state has no leader
I20250112 20:51:39.494230 2140 raft_consensus.cc:491] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.494381 2140 raft_consensus.cc:513] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.495086 2140 leader_election.cc:290] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.495251 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a6bfa15d0fd54baca3f4e1d91de676fe
I20250112 20:51:39.496157 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.496372 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.497129 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a6bfa15d0fd54baca3f4e1d91de676fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6bfa15d0fd54baca3f4e1d91de676fe already in progress: creating tablet
I20250112 20:51:39.498437 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1053991511694aa9b914c54e7f01e35a" } new_table_name: "default.Table_9b0b92a10e3e4a2696b2ca52a9382016" modify_external_catalogs: true
I20250112 20:51:39.498922 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.500741 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b85b9e82f9e847c4acfeecb82201e694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85b9e82f9e847c4acfeecb82201e694 already in progress: creating tablet
I20250112 20:51:39.502800 879 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfa15d0fd54baca3f4e1d91de676fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.503696 2296 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.504351 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a01cef1a596c47309a0b659209428d4b" } new_table_name: "default.Table_1d050efe434c480199f7df0832274841" modify_external_catalogs: true
I20250112 20:51:39.505916 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b5d6bc8b522a4be197bd045c77d4a5f4" } modify_external_catalogs: true
I20250112 20:51:39.506104 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b5d6bc8b522a4be197bd045c77d4a5f4" } modify_external_catalogs: true
I20250112 20:51:39.509088 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d3f2faf0c65b4e5a85d4694428f6c9d5" } new_table_name: "default.Table_2a367254f6944567a9e0efe20de5399b" modify_external_catalogs: true
I20250112 20:51:39.511170 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_857cd22bec1a484eb38f055272a9a9a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.511705 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857cd22bec1a484eb38f055272a9a9a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:39.519366 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a6bfa15d0fd54baca3f4e1d91de676fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6bfa15d0fd54baca3f4e1d91de676fe already in progress: creating tablet
W20250112 20:51:39.530325 710 leader_election.cc:343] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.530797 710 leader_election.cc:343] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.531210 710 leader_election.cc:343] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.531337 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 3b57ec6e137f4ec5b93b6a4c4912fddb
I20250112 20:51:39.532351 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cdb9a415f745cbb62f27763e490d5b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:39.532613 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_875e19c996d5437894e99baaa6fcffbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.533342 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_875e19c996d5437894e99baaa6fcffbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.534917 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b310e4135c2b424fb006a38a60e41066" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:39.539032 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a185654a21340bfb877caebb7b13063 consensus state has no leader
I20250112 20:51:39.543135 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.543112 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8aba8733d71c40e690d75bc533ebddb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.543597 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aba8733d71c40e690d75bc533ebddb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:39.545368 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaa091fe2d3c4f72815f228ac3e39128 consensus state has no leader
I20250112 20:51:39.545629 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 15b009608f514353be8a58408b4abdea from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:39.547358 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 44187.3us
I20250112 20:51:39.547688 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 615d75bc22df45cc876788a266e983cc
W20250112 20:51:39.547937 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a6bfa15d0fd54baca3f4e1d91de676fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6bfa15d0fd54baca3f4e1d91de676fe already in progress: creating tablet
I20250112 20:51:39.548274 454 master_service.cc:946] ReplaceTablet: received request to replace tablet be6751397eab403a8998a6a71820a6f9 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:39.549213 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5ae913686e349849b407e9ec9d6ef0b consensus state has no leader
I20250112 20:51:39.549434 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 60eefdbabc174cd6a67fd12ea7d95071 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:39.551412 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6b2c4ea5f4167896c78b1c64cbb62 already in progress: creating tablet
I20250112 20:51:39.553200 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.553892 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 615d75bc22df45cc876788a266e983cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615d75bc22df45cc876788a266e983cc already in progress: creating tablet
I20250112 20:51:39.554276 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.558868 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.559850 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6b2c4ea5f4167896c78b1c64cbb62 already in progress: creating tablet
I20250112 20:51:39.534341 710 leader_election.cc:304] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.548486 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.562532 2326 raft_consensus.cc:2798] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.562713 2326 raft_consensus.cc:695] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.563024 2326 consensus_queue.cc:237] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.564602 710 leader_election.cc:304] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.565096 2324 raft_consensus.cc:2798] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.565167 2324 raft_consensus.cc:491] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.565224 2324 raft_consensus.cc:3054] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.568094 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet be6751397eab403a8998a6a71820a6f9
I20250112 20:51:39.568518 454 catalog_manager.cc:6596] ReplaceTablet: tablet be6751397eab403a8998a6a71820a6f9 deleted and replaced by tablet 38979539a3ae4d08a398874bf894ac32
I20250112 20:51:39.568953 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b9a0b9c1271a440f95be03d7aff157ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.569664 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9a0b9c1271a440f95be03d7aff157ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.569777 2324 raft_consensus.cc:513] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.570448 2324 leader_election.cc:290] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.571141 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.571559 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.572873 738 tablet_service.cc:1514] Processing DeleteTablet for tablet be6751397eab403a8998a6a71820a6f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 38979539a3ae4d08a398874bf894ac32) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.573022 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c275eb298fde46e7b70911f21c75fffe" } new_table_name: "default.Table_bf9df2a098b246ddaf645cbf3426550a" modify_external_catalogs: true
I20250112 20:51:39.573432 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet be6751397eab403a8998a6a71820a6f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 38979539a3ae4d08a398874bf894ac32) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.574560 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet be6751397eab403a8998a6a71820a6f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be6751397eab403a8998a6a71820a6f9 already in progress: creating tablet
I20250112 20:51:39.577659 1164 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.577806 1164 ts_tablet_manager.cc:1397] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.216s user 0.002s sys 0.003s
W20250112 20:51:39.578423 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 615d75bc22df45cc876788a266e983cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615d75bc22df45cc876788a266e983cc already in progress: creating tablet
I20250112 20:51:39.578431 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c8d24bccf682483aa9b273fc1ce847da from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:39.578576 1164 raft_consensus.cc:357] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.578748 1164 raft_consensus.cc:383] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.578819 1164 raft_consensus.cc:738] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.579013 1164 consensus_queue.cc:260] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:39.580124 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 615d75bc22df45cc876788a266e983cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615d75bc22df45cc876788a266e983cc already in progress: creating tablet
W20250112 20:51:39.580402 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet be6751397eab403a8998a6a71820a6f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be6751397eab403a8998a6a71820a6f9 already in progress: creating tablet
I20250112 20:51:39.581372 1178 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.581557 1178 ts_tablet_manager.cc:1397] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.224s user 0.009s sys 0.041s
I20250112 20:51:39.582783 1178 raft_consensus.cc:357] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.583039 1178 raft_consensus.cc:383] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.583148 1178 raft_consensus.cc:738] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.583207 2296 tablet.cc:1721] T 85897bfec1834c57956a23c5dfc2e853 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.583415 1178 consensus_queue.cc:260] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.584003 1178 ts_tablet_manager.cc:1428] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:39.584360 1178 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.585031 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.586189 1178 tablet_bootstrap.cc:654] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:39.588493 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b85b9e82f9e847c4acfeecb82201e694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85b9e82f9e847c4acfeecb82201e694 already in progress: creating tablet
I20250112 20:51:39.589251 893 tablet_service.cc:1514] Processing DeleteTablet for tablet be6751397eab403a8998a6a71820a6f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 38979539a3ae4d08a398874bf894ac32) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:39.589963 710 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b310e4135c2b424fb006a38a60e41066
I20250112 20:51:39.591526 1164 ts_tablet_manager.cc:1428] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250112 20:51:39.591948 2339 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:39.593209 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet be6751397eab403a8998a6a71820a6f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be6751397eab403a8998a6a71820a6f9 already in progress: creating tablet
I20250112 20:51:39.593631 2350 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.601642 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c8d24bccf682483aa9b273fc1ce847da
I20250112 20:51:39.602141 447 catalog_manager.cc:6596] ReplaceTablet: tablet c8d24bccf682483aa9b273fc1ce847da deleted and replaced by tablet 370ad081bf784808b2ab44cad8d80959
W20250112 20:51:39.602789 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b1565c7487c428f8f963aca406263fe consensus state has no leader
I20250112 20:51:39.603533 1178 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.603699 1178 ts_tablet_manager.cc:1397] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250112 20:51:39.604382 1178 raft_consensus.cc:357] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.604620 1178 raft_consensus.cc:383] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.604707 1178 raft_consensus.cc:738] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.604676 983 leader_election.cc:304] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:39.605124 2140 raft_consensus.cc:2743] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.605046 1178 consensus_queue.cc:260] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.605510 1178 ts_tablet_manager.cc:1428] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:39.605742 738 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.605809 1178 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:39.606263 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:39.606426 847 leader_election.cc:304] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:39.612015 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:39.612763 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c8d24bccf682483aa9b273fc1ce847da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8d24bccf682483aa9b273fc1ce847da already in progress: creating tablet
W20250112 20:51:39.613967 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c8d24bccf682483aa9b273fc1ce847da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8d24bccf682483aa9b273fc1ce847da already in progress: creating tablet
I20250112 20:51:39.618489 2350 tablet.cc:1721] T 85897bfec1834c57956a23c5dfc2e853 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.623833 1163 tablet_bootstrap.cc:492] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.623963 1163 ts_tablet_manager.cc:1397] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.238s user 0.023s sys 0.038s
I20250112 20:51:39.624644 1163 raft_consensus.cc:357] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.624881 1163 raft_consensus.cc:383] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.624938 1163 raft_consensus.cc:738] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.625121 1163 consensus_queue.cc:260] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.625591 1163 ts_tablet_manager.cc:1428] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:39.625828 1163 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.607311 1178 tablet_bootstrap.cc:654] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.627316 1163 tablet_bootstrap.cc:654] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.627902 1164 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:39.628749 2108 raft_consensus.cc:2743] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.630000 1164 tablet_bootstrap.cc:654] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.629977 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.630153 912 raft_consensus.cc:3054] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.633111 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8fae6a2fd2e041f6953157f01d134e04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.633683 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fae6a2fd2e041f6953157f01d134e04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.634367 912 raft_consensus.cc:2463] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.635146 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:39.636469 2296 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.637280 1178 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.637224 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c076efa279bf4b74bcb85f1f0d355bed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.637393 1178 ts_tablet_manager.cc:1397] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.002s
W20250112 20:51:39.637688 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c076efa279bf4b74bcb85f1f0d355bed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.638032 1178 raft_consensus.cc:357] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.638173 1178 raft_consensus.cc:383] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.638211 1178 raft_consensus.cc:738] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.638372 1178 consensus_queue.cc:260] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.638845 1178 ts_tablet_manager.cc:1428] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:39.639062 1178 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.640172 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f962e711e3ff42a69fa1d2ac1b9ec27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.640549 1178 tablet_bootstrap.cc:654] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:39.640630 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f962e711e3ff42a69fa1d2ac1b9ec27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.644759 1164 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.646011 1164 ts_tablet_manager.cc:1397] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250112 20:51:39.646283 893 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:39.647002 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c8d24bccf682483aa9b273fc1ce847da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8d24bccf682483aa9b273fc1ce847da already in progress: creating tablet
I20250112 20:51:39.647379 1164 raft_consensus.cc:357] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.647578 1164 raft_consensus.cc:383] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.647648 1164 raft_consensus.cc:738] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.655011 1164 consensus_queue.cc:260] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.655501 1164 ts_tablet_manager.cc:1428] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250112 20:51:39.656242 1163 tablet_bootstrap.cc:492] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.656383 1163 ts_tablet_manager.cc:1397] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.005s
W20250112 20:51:39.656961 710 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.657161 710 leader_election.cc:304] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:39.657392 1163 raft_consensus.cc:357] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.657586 1163 raft_consensus.cc:383] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.657645 1163 raft_consensus.cc:738] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.657737 2326 raft_consensus.cc:2743] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:39.657840 1163 consensus_queue.cc:260] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.658331 1163 ts_tablet_manager.cc:1428] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:39.658591 1163 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.658957 1178 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.659166 1178 ts_tablet_manager.cc:1397] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20250112 20:51:39.660207 1163 tablet_bootstrap.cc:654] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.660655 1178 raft_consensus.cc:357] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.660943 1178 raft_consensus.cc:383] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.661021 1178 raft_consensus.cc:738] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.661329 1178 consensus_queue.cc:260] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.662113 1178 ts_tablet_manager.cc:1428] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:39.662987 1178 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:39.666667 710 leader_election.cc:343] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:39.666993 710 consensus_peers.cc:487] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:39.670172 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 38cdb9a415f745cbb62f27763e490d5b (table default.Table_5d0ef950ae7941faa216af8392ab9a72 [id=d20f36b81b8d4a1ba8c0c792e1a997c7]). Expected version 1 got 0
I20250112 20:51:39.670276 1178 tablet_bootstrap.cc:654] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.670406 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet a218ef5825424b1b84c55b38b9b9023d (table default.Table_df950a6830fa45b1a8f21feda53ea0ea [id=05211d8527144710860dc2aca1ac76a3]). Expected version 1 got 0
I20250112 20:51:39.670475 2339 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.670569 448 catalog_manager.cc:5526] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 1 to 2, leader changed from aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1) to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 2 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.671027 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:39.671239 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet f5aca1ff0fd541bfa96bf7f279fa4e2d (table default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2]). Expected version 1 got 0
I20250112 20:51:39.671376 448 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 85897bfec1834c57956a23c5dfc2e853 (table default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7]). Expected version 1 got 0
I20250112 20:51:39.682415 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 15b009608f514353be8a58408b4abdea
I20250112 20:51:39.682881 452 catalog_manager.cc:6596] ReplaceTablet: tablet 15b009608f514353be8a58408b4abdea deleted and replaced by tablet 29809c91e33a491a8503d7951135b15d
I20250112 20:51:39.691156 752 tablet_service.cc:1467] Processing CreateTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 (DEFAULT_TABLE table=default.Table_6f7b30f2842340de9879b5e275a9c8a4 [id=5627a77211e049cda45ea285dc3ff7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.691610 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ccaeaf38c41a6ab26f65411d31418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.691156 737 tablet_service.cc:1467] Processing CreateTablet for tablet 140276b5b8c14512bb28b864bbbc3843 (DEFAULT_TABLE table=default.Table_aeb009faaf764c289ecacebf52f2fcdd [id=e6e49643abd84cb6b8ab0338bb8b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.691879 738 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0786594542416cb67c4112921943b3 (DEFAULT_TABLE table=default.Table_90bbde70dc014c87be25b842e16d44d0 [id=514e3516abe84e25862095eb98de758d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.692147 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140276b5b8c14512bb28b864bbbc3843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.692312 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0786594542416cb67c4112921943b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.692536 747 tablet_service.cc:1467] Processing CreateTablet for tablet 25fec75d34da4d75800fbadedb1b1aa4 (DEFAULT_TABLE table=default.Table_1b79e8a8774d47a6a96ca46ce2427b18 [id=b92a3d8f86834af99415beb5c234d879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.693377 753 tablet_service.cc:1467] Processing CreateTablet for tablet 91795c27f0c84d3781789fd5242d6c3a (DEFAULT_TABLE table=default.Table_e25daa065bc34cef93655a371805ccf3 [id=d8aac7f256ae42f8a65b2c3f6ad455dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.693629 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91795c27f0c84d3781789fd5242d6c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.694124 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f (DEFAULT_TABLE table=default.Table_d410c058166640f08a520ceaec274a90 [id=79fe8afad41a4a4e88c4086bf2a6daac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.694574 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_424524dba512432681b74ca38ea110ff" } modify_external_catalogs: true
I20250112 20:51:39.694810 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_424524dba512432681b74ca38ea110ff" } modify_external_catalogs: true
I20250112 20:51:39.695854 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c16f29e9cecc475a82c8f35e9b2e27ed" } new_table_name: "default.Table_a56ab4e8789f432ba82943e7830afea7" modify_external_catalogs: true
I20250112 20:51:39.695971 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b009608f514353be8a58408b4abdea with delete_type TABLET_DATA_DELETED (replaced by tablet 29809c91e33a491a8503d7951135b15d) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.696620 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 15b009608f514353be8a58408b4abdea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15b009608f514353be8a58408b4abdea already in progress: creating tablet
I20250112 20:51:39.697193 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fec75d34da4d75800fbadedb1b1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.697430 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a9962f17769a4deaab73b90205bb15bd" } modify_external_catalogs: true
I20250112 20:51:39.697600 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a9962f17769a4deaab73b90205bb15bd" } modify_external_catalogs: true
I20250112 20:51:39.698511 2108 raft_consensus.cc:491] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.698637 2108 raft_consensus.cc:513] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.699268 2108 leader_election.cc:290] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.699838 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6a19ee7a4e4afab12870e48ee8df7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.700300 1178 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:39.700482 1178 ts_tablet_manager.cc:1397] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.005s
I20250112 20:51:39.700663 754 tablet_service.cc:1467] Processing CreateTablet for tablet efe90c40f96c4736954e22a127ced1b7 (DEFAULT_TABLE table=default.Table_22f40a2d9f084970b9d2152f401edd49 [id=3834d8cb81874d87ac21c355669e2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.701017 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe90c40f96c4736954e22a127ced1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.701845 1178 raft_consensus.cc:357] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.702142 1178 raft_consensus.cc:383] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.702214 1178 raft_consensus.cc:738] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.702523 1178 consensus_queue.cc:260] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.702877 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 (DEFAULT_TABLE table=default.Table_861997ae234b444fae5e7160ed43a3e7 [id=944e81d78b914f77b29a99fd3b44c81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.703212 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0866871b9e1542e8ab30505c68f7dea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.703588 1178 ts_tablet_manager.cc:1428] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:39.704154 1178 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.704804 453 catalog_manager.cc:5526] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.705216 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]). Expected version 1 got 0
I20250112 20:51:39.705662 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:39.705974 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 1 got 0
I20250112 20:51:39.706175 453 catalog_manager.cc:5526] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.706631 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 85897bfec1834c57956a23c5dfc2e853 (table default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7]). Expected version 1 got 0
I20250112 20:51:39.706979 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:39.707193 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:39.707350 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:39.708307 1178 tablet_bootstrap.cc:654] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.716150 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 902fc636a0494462ae1e73b0cefc5dfe
I20250112 20:51:39.717005 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_45f57bdcc8a24f6695242226c7422b8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.717562 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45f57bdcc8a24f6695242226c7422b8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.717605 2326 raft_consensus.cc:491] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.717734 2326 raft_consensus.cc:513] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.718345 2326 leader_election.cc:290] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.719223 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 60eefdbabc174cd6a67fd12ea7d95071
I20250112 20:51:39.719640 2108 raft_consensus.cc:491] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.719770 449 catalog_manager.cc:6596] ReplaceTablet: tablet 60eefdbabc174cd6a67fd12ea7d95071 deleted and replaced by tablet b29fe38d2fb34ede920f5c497b585110
I20250112 20:51:39.719775 2108 raft_consensus.cc:513] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.720502 2108 leader_election.cc:290] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.721184 710 leader_election.cc:304] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:39.721457 2326 raft_consensus.cc:2798] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.721678 2326 raft_consensus.cc:695] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.657402 1164 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:39.722905 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 18664fc1f93f4d48a75223c80da29b7a
I20250112 20:51:39.723755 2324 consensus_queue.cc:1035] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:39.725337 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_10c19a0eede64f589d42d79a287f52f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.725924 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10c19a0eede64f589d42d79a287f52f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.726737 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64887b3578284cb286ee122ab8c626c4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.726972 912 raft_consensus.cc:2463] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.728046 746 tablet_service.cc:1467] Processing CreateTablet for tablet 6921185b545244978f8fdeafd1fb6c5e (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.728420 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6921185b545244978f8fdeafd1fb6c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.729017 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 18664fc1f93f4d48a75223c80da29b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.730556 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 18664fc1f93f4d48a75223c80da29b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18664fc1f93f4d48a75223c80da29b7a already in progress: creating tablet
I20250112 20:51:39.732628 741 tablet_service.cc:1467] Processing CreateTablet for tablet 72252e88e6664bab850663236d280987 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.733091 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72252e88e6664bab850663236d280987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.733893 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 with delete_type TABLET_DATA_DELETED (replaced by tablet b29fe38d2fb34ede920f5c497b585110) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.734285 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_16d626d49adf42eea0a98976db7a45b0" } modify_external_catalogs: true
W20250112 20:51:39.734483 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 60eefdbabc174cd6a67fd12ea7d95071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60eefdbabc174cd6a67fd12ea7d95071 already in progress: creating tablet
I20250112 20:51:39.734532 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_16d626d49adf42eea0a98976db7a45b0" } modify_external_catalogs: true
I20250112 20:51:39.736407 2326 consensus_queue.cc:237] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.741372 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0140e608c30d4711841b87316d32f6c3" } new_table_name: "default.Table_065c3ecd3a16435f841e67981156488c" modify_external_catalogs: true
I20250112 20:51:39.748772 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1b4f716257454c0bbc3266a5706d97e4" } new_table_name: "default.Table_7ca72a25393a40d9bdbbb1d078b7cff2" modify_external_catalogs: true
I20250112 20:51:39.751601 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e2d706e7614440a1905f8790f025ac29" } modify_external_catalogs: true
I20250112 20:51:39.751883 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e2d706e7614440a1905f8790f025ac29" } modify_external_catalogs: true
I20250112 20:51:39.752521 710 leader_election.cc:304] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.754242 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d410c058166640f08a520ceaec274a90" } new_table_name: "default.Table_00baf585f7a14dd3a9c7998b5956c6e1" modify_external_catalogs: true
I20250112 20:51:39.756546 2324 raft_consensus.cc:2798] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.756628 2324 raft_consensus.cc:491] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.756687 2324 raft_consensus.cc:3054] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.760092 453 catalog_manager.cc:5526] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.760311 744 tablet_service.cc:1467] Processing CreateTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 (DEFAULT_TABLE table=default.Table_19cd3e26ebaa41008374cb9290c3e323 [id=c3da756cc9de43ef951e7a42b7864d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.761075 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_9d078f3dac874426bfcdd9504733991c [id=3861236543364f8cbcd9632b17fca2f2] alter complete (version 1)
I20250112 20:51:39.728046 749 tablet_service.cc:1467] Processing CreateTablet for tablet 12b7db4f265f475e9c8530818918221b (DEFAULT_TABLE table=default.Table_0140e608c30d4711841b87316d32f6c3 [id=ccee75651cb4471faed27f360f675220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.762966 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3f0a808d431a43fb9299fec1dcf6d3b9
W20250112 20:51:39.767402 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 46) took 858 ms (client timeout 1000 ms). Trace:
W20250112 20:51:39.767539 450 rpcz_store.cc:269] 0112 20:51:38.909286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:38.957540 (+ 48254us) service_pool.cc:225] Handling call
0112 20:51:39.013448 (+ 55908us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14f96b8 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:39.767356 (+753908us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:39.434532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:39.434610 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:39.434614 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:39.434617 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:39.434684 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:39.434708 (+ 24us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:39.434712 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:39.434754 (+ 42us) tablet.cc:698] Row locks acquired
0112 20:51:39.434757 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:39.434781 (+ 24us) write_op.cc:270] Start()
0112 20:51:39.434815 (+ 34us) write_op.cc:276] Timestamp: P: 1736715099434725 usec, L: 0
0112 20:51:39.434818 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:39.435000 (+ 182us) log.cc:844] Serialized 527 byte log entry
0112 20:51:39.480953 (+ 45953us) op_driver.cc:464] REPLICATION: finished
0112 20:51:39.481043 (+ 90us) write_op.cc:301] APPLY: starting
0112 20:51:39.481058 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:39.481114 (+ 56us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:39.481143 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:39.481191 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:39.481228 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:39.481239 (+ 11us) write_op.cc:312] APPLY: finished
0112 20:51:39.481309 (+ 70us) log.cc:844] Serialized 23 byte log entry
0112 20:51:39.481375 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:39.481391 (+ 16us) write_op.cc:454] Released schema lock
0112 20:51:39.481399 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":13,"spinlock_wait_cycles":126777088,"child_traces":[["op",{"apply.queue_time_us":43,"apply.run_cpu_time_us":420,"apply.run_wall_time_us":420,"num_ops":1,"prepare.queue_time_us":79787,"prepare.run_cpu_time_us":527,"prepare.run_wall_time_us":527,"replication_time_us":46107,"spinlock_wait_cycles":102656}]]}
I20250112 20:51:39.777531 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_96a40b39750b413e9567ab229a39f9b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.778088 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96a40b39750b413e9567ab229a39f9b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.780794 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_28f569ded7ee456cb79ce58b8adb809d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.781291 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28f569ded7ee456cb79ce58b8adb809d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.782604 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4633e6870ccd4ce5b2458bb22155ba42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.782846 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b7db4f265f475e9c8530818918221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.785120 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cbc22207e9204b73877c74f61ebba8a8
I20250112 20:51:39.785274 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.785580 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.785624 1185 tablet_replica.cc:331] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:39.785883 1185 raft_consensus.cc:2238] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:39.786240 1185 raft_consensus.cc:2267] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.786942 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d587a303a8b44cecacc313b337eb84b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.787457 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d587a303a8b44cecacc313b337eb84b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.788515 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.793915 2059 raft_consensus.cc:491] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.794052 2059 raft_consensus.cc:513] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.794608 2059 leader_election.cc:290] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:39.795300 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12b7db4f265f475e9c8530818918221b consensus state has no leader
I20250112 20:51:39.795734 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8ac2217609124d6ba7936c9268f10af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.796304 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:39.796411 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac2217609124d6ba7936c9268f10af7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.797891 752 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.799798 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.803090 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.803800 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.803565 2324 raft_consensus.cc:513] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.804654 2324 leader_election.cc:290] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.806531 1185 ts_tablet_manager.cc:1905] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:39.808897 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30156185f6104eada87b936fc1829df9 consensus state has no leader
I20250112 20:51:39.809980 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_99673c7bb11c4ff5a0b8879e673d3c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.810571 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99673c7bb11c4ff5a0b8879e673d3c3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.813194 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64887b3578284cb286ee122ab8c626c4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.813545 2355 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.814024 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b009608f514353be8a58408b4abdea with delete_type TABLET_DATA_DELETED (replaced by tablet 29809c91e33a491a8503d7951135b15d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.818392 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf1dc64f1964202bfa8169388a34781" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.818572 912 raft_consensus.cc:2463] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.819182 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64887b3578284cb286ee122ab8c626c4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.819321 912 raft_consensus.cc:3054] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.819674 1185 ts_tablet_manager.cc:1918] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:39.819835 1185 log.cc:1198] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:39.820566 1185 ts_tablet_manager.cc:1939] T 81b932b7e1e34d4ab18aae40047dc038 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:39.821791 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7ab5ecd2698b4952a892135b3593106a
I20250112 20:51:39.822070 912 raft_consensus.cc:2463] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.822240 1185 tablet_replica.cc:331] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:39.822515 1185 raft_consensus.cc:2238] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.822757 1185 raft_consensus.cc:2267] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.823284 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bf1f1d5c2be64a72af1c3c8396cb7180" } new_table_name: "default.Table_9f6a90a2894849e08360909243f6a463" modify_external_catalogs: true
I20250112 20:51:39.823660 1185 ts_tablet_manager.cc:1905] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.825546 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0786594542416cb67c4112921943b3 (DEFAULT_TABLE table=default.Table_90bbde70dc014c87be25b842e16d44d0 [id=514e3516abe84e25862095eb98de758d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.825947 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0786594542416cb67c4112921943b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.829094 1164 tablet_bootstrap.cc:654] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.832486 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 140276b5b8c14512bb28b864bbbc3843 (DEFAULT_TABLE table=default.Table_aeb009faaf764c289ecacebf52f2fcdd [id=e6e49643abd84cb6b8ab0338bb8b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.832839 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140276b5b8c14512bb28b864bbbc3843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.833268 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 81b932b7e1e34d4ab18aae40047dc038 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
I20250112 20:51:39.833599 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab5ecd2698b4952a892135b3593106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.834203 1163 tablet_bootstrap.cc:492] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:39.834322 1163 ts_tablet_manager.cc:1397] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.176s user 0.002s sys 0.003s
I20250112 20:51:39.835111 1163 raft_consensus.cc:357] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.835304 1163 raft_consensus.cc:383] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.835357 1163 raft_consensus.cc:738] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.835561 1163 consensus_queue.cc:260] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.836272 1163 ts_tablet_manager.cc:1428] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:39.836519 1163 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:39.837042 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f (DEFAULT_TABLE table=default.Table_d410c058166640f08a520ceaec274a90 [id=79fe8afad41a4a4e88c4086bf2a6daac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.837394 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6a19ee7a4e4afab12870e48ee8df7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.838210 1163 tablet_bootstrap.cc:654] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.839978 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944b71897bcf4304bd9f7bb7adef34e5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.840413 2197 raft_consensus.cc:491] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.840504 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.840561 2197 raft_consensus.cc:513] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.841157 2197 leader_election.cc:290] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.842140 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 91795c27f0c84d3781789fd5242d6c3a (DEFAULT_TABLE table=default.Table_e25daa065bc34cef93655a371805ccf3 [id=d8aac7f256ae42f8a65b2c3f6ad455dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.842424 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91795c27f0c84d3781789fd5242d6c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.854030 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 (DEFAULT_TABLE table=default.Table_6f7b30f2842340de9879b5e275a9c8a4 [id=5627a77211e049cda45ea285dc3ff7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.854401 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ccaeaf38c41a6ab26f65411d31418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.854597 2326 raft_consensus.cc:491] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:39.854758 2326 raft_consensus.cc:513] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.855391 2326 leader_election.cc:290] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.856438 1164 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:39.856985 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:39.857339 1164 ts_tablet_manager.cc:1397] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.200s user 0.004s sys 0.002s
I20250112 20:51:39.858181 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:39.858358 913 raft_consensus.cc:2463] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:39.857517 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3846863c74f64923bda6e5a19e7f2770 (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.860394 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944b71897bcf4304bd9f7bb7adef34e5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:39.860558 770 raft_consensus.cc:2463] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:39.860584 1029 tablet_service.cc:1467] Processing CreateTablet for tablet efe90c40f96c4736954e22a127ced1b7 (DEFAULT_TABLE table=default.Table_22f40a2d9f084970b9d2152f401edd49 [id=3834d8cb81874d87ac21c355669e2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.861095 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe90c40f96c4736954e22a127ced1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.861590 710 leader_election.cc:304] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.861137 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:39.864121 770 raft_consensus.cc:2463] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:39.868410 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 3846863c74f64923bda6e5a19e7f2770 (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.868839 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846863c74f64923bda6e5a19e7f2770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.870658 1185 ts_tablet_manager.cc:1918] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.871322 1185 log.cc:1198] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b310e4135c2b424fb006a38a60e41066
I20250112 20:51:39.871955 710 leader_election.cc:304] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.872251 2324 raft_consensus.cc:2798] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.872359 2324 raft_consensus.cc:491] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.872464 2324 raft_consensus.cc:3054] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.873171 2326 raft_consensus.cc:2798] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.873590 2326 raft_consensus.cc:695] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.873855 710 leader_election.cc:304] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.873034 1185 ts_tablet_manager.cc:1939] T b310e4135c2b424fb006a38a60e41066 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:39.874361 2059 raft_consensus.cc:2798] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.874774 2059 raft_consensus.cc:491] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.874747 2339 tablet.cc:1721] T 99ba8529a287434b9ba78e0b6fa7817e P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.874854 2059 raft_consensus.cc:3054] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.876587 2296 tablet.cc:1721] T 99ba8529a287434b9ba78e0b6fa7817e P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.877573 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 12b7db4f265f475e9c8530818918221b (DEFAULT_TABLE table=default.Table_0140e608c30d4711841b87316d32f6c3 [id=ccee75651cb4471faed27f360f675220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.877578 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 6921185b545244978f8fdeafd1fb6c5e (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.878245 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 (DEFAULT_TABLE table=default.Table_19cd3e26ebaa41008374cb9290c3e323 [id=c3da756cc9de43ef951e7a42b7864d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.878301 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.878839 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b7db4f265f475e9c8530818918221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.879015 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6921185b545244978f8fdeafd1fb6c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.879184 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 18664fc1f93f4d48a75223c80da29b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.879616 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4633e6870ccd4ce5b2458bb22155ba42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.886826 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.887101 1184 tablet_replica.cc:331] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.887202 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 with delete_type TABLET_DATA_DELETED (replaced by tablet b29fe38d2fb34ede920f5c497b585110) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.887277 1184 raft_consensus.cc:2238] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.887411 1184 raft_consensus.cc:2267] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.887926 1184 ts_tablet_manager.cc:1905] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.893867 1184 ts_tablet_manager.cc:1918] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.894073 1184 log.cc:1198] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3f0a808d431a43fb9299fec1dcf6d3b9
I20250112 20:51:39.894415 1184 ts_tablet_manager.cc:1939] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:39.894683 1164 raft_consensus.cc:357] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.895082 1164 raft_consensus.cc:383] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.895161 1164 raft_consensus.cc:738] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.895383 1164 consensus_queue.cc:260] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.896064 1164 ts_tablet_manager.cc:1428] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20250112 20:51:39.895911 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 25fec75d34da4d75800fbadedb1b1aa4 (DEFAULT_TABLE table=default.Table_1b79e8a8774d47a6a96ca46ce2427b18 [id=b92a3d8f86834af99415beb5c234d879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.896462 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fec75d34da4d75800fbadedb1b1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:39.899410 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 58905.5us
I20250112 20:51:39.899885 847 leader_election.cc:304] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.900327 2197 raft_consensus.cc:2798] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.900425 2197 raft_consensus.cc:491] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.900506 2197 raft_consensus.cc:3054] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.900637 847 leader_election.cc:304] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.900923 2108 raft_consensus.cc:2798] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:39.901001 2108 raft_consensus.cc:491] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.901087 2108 raft_consensus.cc:3054] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.903317 2197 raft_consensus.cc:513] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.903487 2108 raft_consensus.cc:513] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.904153 2197 leader_election.cc:290] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:39.904316 2108 leader_election.cc:290] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:39.904894 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b532068d2374b59a3c46525db7c50db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:39.905288 1184 tablet_replica.cc:331] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:39.905560 1184 raft_consensus.cc:2238] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.905807 1184 raft_consensus.cc:2267] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.906394 1184 ts_tablet_manager.cc:1905] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.912668 1184 ts_tablet_manager.cc:1918] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:39.912596 2324 raft_consensus.cc:513] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.913305 2324 leader_election.cc:290] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:39.913901 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846863c74f64923bda6e5a19e7f2770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.916122 2059 raft_consensus.cc:513] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.916635 2059 leader_election.cc:290] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:39.917634 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c6a19ee7a4e4afab12870e48ee8df7f consensus state has no leader
I20250112 20:51:39.919977 2326 consensus_queue.cc:237] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:39.926849 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf1dc64f1964202bfa8169388a34781" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.927075 912 raft_consensus.cc:3054] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.928771 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b310e4135c2b424fb006a38a60e41066 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
I20250112 20:51:39.929203 1185 tablet_replica.cc:331] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:39.929729 912 raft_consensus.cc:2463] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:39.930778 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:39.930969 912 raft_consensus.cc:3054] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.930421 1178 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
W20250112 20:51:39.931300 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 15b009608f514353be8a58408b4abdea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15b009608f514353be8a58408b4abdea already in progress: creating tablet
W20250112 20:51:39.931650 847 leader_election.cc:343] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.931376 1164 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:39.932224 847 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.931639 1185 raft_consensus.cc:2238] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:39.933862 710 leader_election.cc:304] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:39.934214 2059 raft_consensus.cc:2798] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.934332 2059 raft_consensus.cc:695] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.934551 2059 consensus_queue.cc:237] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:39.938755 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 902fc636a0494462ae1e73b0cefc5dfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902fc636a0494462ae1e73b0cefc5dfe already in progress: creating tablet
I20250112 20:51:39.939159 912 raft_consensus.cc:2463] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:39.939322 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 18664fc1f93f4d48a75223c80da29b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18664fc1f93f4d48a75223c80da29b7a already in progress: creating tablet
I20250112 20:51:39.940155 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b009608f514353be8a58408b4abdea with delete_type TABLET_DATA_DELETED (replaced by tablet 29809c91e33a491a8503d7951135b15d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.942574 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0786594542416cb67c4112921943b3 (DEFAULT_TABLE table=default.Table_90bbde70dc014c87be25b842e16d44d0 [id=514e3516abe84e25862095eb98de758d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.942997 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0786594542416cb67c4112921943b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:39.943126 710 leader_election.cc:343] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:39.932600 1178 ts_tablet_manager.cc:1397] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.229s user 0.018s sys 0.029s
I20250112 20:51:39.946681 1185 raft_consensus.cc:2267] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:39.947875 710 leader_election.cc:304] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:39.949635 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 30156185f6104eada87b936fc1829df9 consensus state has no leader
I20250112 20:51:39.950513 2326 raft_consensus.cc:2798] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.950822 2326 raft_consensus.cc:695] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:39.950958 893 tablet_service.cc:1467] Processing CreateTablet for tablet 25fec75d34da4d75800fbadedb1b1aa4 (DEFAULT_TABLE table=default.Table_1b79e8a8774d47a6a96ca46ce2427b18 [id=b92a3d8f86834af99415beb5c234d879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.951056 2326 consensus_queue.cc:237] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:39.951402 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fec75d34da4d75800fbadedb1b1aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.957106 1164 tablet_bootstrap.cc:654] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.957741 1185 ts_tablet_manager.cc:1905] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:39.958330 889 tablet_service.cc:1467] Processing CreateTablet for tablet 140276b5b8c14512bb28b864bbbc3843 (DEFAULT_TABLE table=default.Table_aeb009faaf764c289ecacebf52f2fcdd [id=e6e49643abd84cb6b8ab0338bb8b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.958683 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140276b5b8c14512bb28b864bbbc3843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.958863 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (table default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]) successfully deleted
W20250112 20:51:39.960975 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 60eefdbabc174cd6a67fd12ea7d95071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60eefdbabc174cd6a67fd12ea7d95071 already in progress: creating tablet
I20250112 20:51:39.961062 1178 raft_consensus.cc:357] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.961244 1178 raft_consensus.cc:383] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.961305 1178 raft_consensus.cc:738] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.961694 1178 consensus_queue.cc:260] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:39.961853 880 tablet_service.cc:1467] Processing CreateTablet for tablet 91795c27f0c84d3781789fd5242d6c3a (DEFAULT_TABLE table=default.Table_e25daa065bc34cef93655a371805ccf3 [id=d8aac7f256ae42f8a65b2c3f6ad455dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.962218 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91795c27f0c84d3781789fd5242d6c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.965016 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f (DEFAULT_TABLE table=default.Table_d410c058166640f08a520ceaec274a90 [id=79fe8afad41a4a4e88c4086bf2a6daac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.965329 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6a19ee7a4e4afab12870e48ee8df7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.969066 878 tablet_service.cc:1467] Processing CreateTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 (DEFAULT_TABLE table=default.Table_6f7b30f2842340de9879b5e275a9c8a4 [id=5627a77211e049cda45ea285dc3ff7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.969435 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532ccaeaf38c41a6ab26f65411d31418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.969410 886 tablet_service.cc:1467] Processing CreateTablet for tablet 72252e88e6664bab850663236d280987 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.969687 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72252e88e6664bab850663236d280987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.969873 879 tablet_service.cc:1467] Processing CreateTablet for tablet efe90c40f96c4736954e22a127ced1b7 (DEFAULT_TABLE table=default.Table_22f40a2d9f084970b9d2152f401edd49 [id=3834d8cb81874d87ac21c355669e2e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.970160 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efe90c40f96c4736954e22a127ced1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.970503 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b9e4f2b60aef4766b60f2655988f3a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.971127 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9e4f2b60aef4766b60f2655988f3a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.972429 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ecd375c288164374b6c8e4927fd9bc32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:39.973131 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 (DEFAULT_TABLE table=default.Table_861997ae234b444fae5e7160ed43a3e7 [id=944e81d78b914f77b29a99fd3b44c81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.973438 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0866871b9e1542e8ab30505c68f7dea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.973371 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7afee759e98341ce855fc733cb16a5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:39.973785 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afee759e98341ce855fc733cb16a5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:39.975024 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecd375c288164374b6c8e4927fd9bc32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.976996 874 tablet_service.cc:1467] Processing CreateTablet for tablet 3846863c74f64923bda6e5a19e7f2770 (DEFAULT_TABLE table=default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.977319 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846863c74f64923bda6e5a19e7f2770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.962218 1178 ts_tablet_manager.cc:1428] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250112 20:51:39.978470 1178 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:39.981007 888 tablet_service.cc:1467] Processing CreateTablet for tablet 12b7db4f265f475e9c8530818918221b (DEFAULT_TABLE table=default.Table_0140e608c30d4711841b87316d32f6c3 [id=ccee75651cb4471faed27f360f675220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.981318 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b7db4f265f475e9c8530818918221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.984586 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6921185b545244978f8fdeafd1fb6c5e (DEFAULT_TABLE table=default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.984932 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6921185b545244978f8fdeafd1fb6c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.988217 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41" } modify_external_catalogs: true
I20250112 20:51:39.988409 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_10b70fbe941d4ea8b6bedd9c36e0cc41" } modify_external_catalogs: true
I20250112 20:51:39.988885 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 902fc636a0494462ae1e73b0cefc5dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:39.988950 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_133503fd7b744d8f8e3db0843e4ac9e2" } modify_external_catalogs: true
I20250112 20:51:39.989128 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_133503fd7b744d8f8e3db0843e4ac9e2" } modify_external_catalogs: true
I20250112 20:51:39.991932 1178 tablet_bootstrap.cc:654] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.005808 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 (DEFAULT_TABLE table=default.Table_19cd3e26ebaa41008374cb9290c3e323 [id=c3da756cc9de43ef951e7a42b7864d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.006237 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4633e6870ccd4ce5b2458bb22155ba42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.006834 1185 ts_tablet_manager.cc:1918] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.007970 1185 log.cc:1198] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3f0a808d431a43fb9299fec1dcf6d3b9
I20250112 20:51:40.014056 1185 ts_tablet_manager.cc:1939] T 3f0a808d431a43fb9299fec1dcf6d3b9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.015147 2238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Bootstrap replayed 1/1 log segments. Stats: ops{read=376 overwritten=0 applied=376 ignored=0} inserts{seen=347 ignored=0} mutations{seen=406 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:40.015108 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ca3bce77c81b44b98c37ce00f1536286" } new_table_name: "default.Table_d7b9b38db32f43d2af6745ffdddec421" modify_external_catalogs: true
I20250112 20:51:40.015766 2238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Bootstrap complete.
I20250112 20:51:40.021876 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (table default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]) successfully deleted
I20250112 20:51:40.022951 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 60eefdbabc174cd6a67fd12ea7d95071 with delete_type TABLET_DATA_DELETED (replaced by tablet b29fe38d2fb34ede920f5c497b585110) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:40.024391 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 902fc636a0494462ae1e73b0cefc5dfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902fc636a0494462ae1e73b0cefc5dfe already in progress: creating tablet
W20250112 20:51:40.026965 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cbc22207e9204b73877c74f61ebba8a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbc22207e9204b73877c74f61ebba8a8 already in progress: creating tablet
I20250112 20:51:40.026934 2238 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:40.028061 2238 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5c09e899184de6b770dca873501062, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.028098 1185 tablet_replica.cc:331] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.028894 2238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 376, Last appended: 1.376, Last appended by leader: 376, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:40.030277 1185 raft_consensus.cc:2238] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.030984 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 18664fc1f93f4d48a75223c80da29b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.032073 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.032420 1186 tablet_replica.cc:331] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.032672 1186 raft_consensus.cc:2238] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.032719 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.032871 1186 raft_consensus.cc:2267] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.032748 1185 pending_rounds.cc:62] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:40.033564 1186 ts_tablet_manager.cc:1905] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.033274 1185 pending_rounds.cc:69] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585046359601152 op_type: NO_OP noop_request { }
I20250112 20:51:40.035709 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:40.036118 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 20:51:40.036561 710 consensus_peers.cc:487] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:40.038542 2238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 20:51:40.044298 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51d2a44677aa492782c7c805300ed6a5 consensus state has no leader
I20250112 20:51:40.034600 1185 raft_consensus.cc:2883] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:40.044869 1185 raft_consensus.cc:2267] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.045768 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_07581b86150545baa24899a6b2452aee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:40.046094 1185 ts_tablet_manager.cc:1905] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:40.046357 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07581b86150545baa24899a6b2452aee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.049913 1164 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.050065 1164 ts_tablet_manager.cc:1397] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.007s
I20250112 20:51:40.050819 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.050874 1164 raft_consensus.cc:357] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.051090 1164 raft_consensus.cc:383] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.051151 1164 raft_consensus.cc:738] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.051390 1164 consensus_queue.cc:260] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.052090 1164 ts_tablet_manager.cc:1428] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 20:51:40.053128 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 15b009608f514353be8a58408b4abdea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15b009608f514353be8a58408b4abdea already in progress: creating tablet
I20250112 20:51:39.928272 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.053267 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.053314 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:40.053467 774 raft_consensus.cc:3054] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.053598 1184 log.cc:1198] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3b532068d2374b59a3c46525db7c50db
I20250112 20:51:40.053926 1184 ts_tablet_manager.cc:1939] T 3b532068d2374b59a3c46525db7c50db P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:40.054342 1041 raft_consensus.cc:3054] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.054572 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d384628ba9b4f81b768eda4f796632b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:39.930298 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab5ecd2698b4952a892135b3593106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.055050 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3b532068d2374b59a3c46525db7c50db (table default.Table_0fd03aba45a64415a098af438c178bc5 [id=e7bc497aaf164b749919f352657861d0]) successfully deleted
I20250112 20:51:39.936646 1163 tablet_bootstrap.cc:492] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.055325 1163 ts_tablet_manager.cc:1397] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.219s user 0.004s sys 0.000s
I20250112 20:51:40.056061 774 raft_consensus.cc:2463] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:40.056085 1163 raft_consensus.cc:357] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.056305 1163 raft_consensus.cc:383] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.056360 1163 raft_consensus.cc:738] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.056557 1163 consensus_queue.cc:260] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.057054 1163 ts_tablet_manager.cc:1428] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:40.056922 1040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944b71897bcf4304bd9f7bb7adef34e5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.057605 1163 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.057898 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944b71897bcf4304bd9f7bb7adef34e5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:40.058038 775 raft_consensus.cc:3054] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.058240 1164 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.058310 2326 consensus_queue.cc:1035] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.059168 1163 tablet_bootstrap.cc:654] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.934919 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 72252e88e6664bab850663236d280987 (DEFAULT_TABLE table=default.Table_6e7ee1324dba4c9398f4b88928f27d86 [id=862dd35c5e5e4d2eb7ff42928adb9f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.060207 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72252e88e6664bab850663236d280987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.060431 775 raft_consensus.cc:2463] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:40.060704 2339 tablet.cc:1721] T 85897bfec1834c57956a23c5dfc2e853 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:39.939883 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 (DEFAULT_TABLE table=default.Table_861997ae234b444fae5e7160ed43a3e7 [id=944e81d78b914f77b29a99fd3b44c81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.964977 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf1dc64f1964202bfa8169388a34781" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:39.997504 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64887b3578284cb286ee122ab8c626c4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.005195 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.014050 1049 raft_consensus.cc:3054] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.030432 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf1dc64f1964202bfa8169388a34781" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.034227 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd46a422093495ba8b5b03039874a32" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.068894 2140 raft_consensus.cc:491] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.069027 2140 raft_consensus.cc:513] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.049716 878 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.069660 2140 leader_election.cc:290] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.049827 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.050279 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.070225 1164 tablet_bootstrap.cc:654] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.050742 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab5ecd2698b4952a892135b3593106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.070382 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.070470 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79ce3118cbf49e287c93a76398521da" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.044345 1186 ts_tablet_manager.cc:1918] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.071143 983 leader_election.cc:304] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:40.071273 1186 log.cc:1198] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4bd46a422093495ba8b5b03039874a32
I20250112 20:51:40.071624 1186 ts_tablet_manager.cc:1939] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:40.071666 2140 raft_consensus.cc:2743] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:40.072098 1186 tablet_replica.cc:331] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.072146 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0866871b9e1542e8ab30505c68f7dea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.072377 1186 raft_consensus.cc:2238] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.072633 1186 raft_consensus.cc:2267] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.075523 1184 tablet_replica.cc:331] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:40.075825 1184 raft_consensus.cc:2238] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.076128 1184 raft_consensus.cc:2267] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.076874 1184 ts_tablet_manager.cc:1905] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.080011 2326 raft_consensus.cc:491] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.080278 2326 raft_consensus.cc:513] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:40.080557 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cbc22207e9204b73877c74f61ebba8a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbc22207e9204b73877c74f61ebba8a8 already in progress: creating tablet
I20250112 20:51:40.081049 2326 leader_election.cc:290] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:40.081632 710 leader_election.cc:343] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.081884 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6d5d425e6a493daa46900cbd8db166" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:40.082000 710 leader_election.cc:343] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.082079 913 raft_consensus.cc:2463] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:40.082264 710 leader_election.cc:343] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.082516 710 leader_election.cc:343] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.082774 710 leader_election.cc:343] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.083050 710 leader_election.cc:304] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.083484 2326 raft_consensus.cc:2798] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.083586 2326 raft_consensus.cc:491] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.083690 2326 raft_consensus.cc:3054] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.050824 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.086761 2326 raft_consensus.cc:513] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.087517 2326 leader_election.cc:290] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.088346 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6d5d425e6a493daa46900cbd8db166" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.088516 913 raft_consensus.cc:3054] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.091182 913 raft_consensus.cc:2463] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.091655 710 leader_election.cc:304] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.092455 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6d5d425e6a493daa46900cbd8db166" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.092667 2059 raft_consensus.cc:2798] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.092823 2059 raft_consensus.cc:695] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.093109 2059 consensus_queue.cc:237] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:40.094422 710 consensus_peers.cc:487] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:40.094826 710 leader_election.cc:343] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.099210 1186 ts_tablet_manager.cc:1905] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.100515 2326 raft_consensus.cc:491] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.100665 2326 raft_consensus.cc:513] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.101266 2326 leader_election.cc:290] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.101897 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829e9dd5854961b5402b5710624612" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.101912 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6d5d425e6a493daa46900cbd8db166" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.102111 912 raft_consensus.cc:2463] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:40.102479 710 leader_election.cc:304] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.103073 2208 raft_consensus.cc:2798] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.103191 2208 raft_consensus.cc:491] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.103292 2208 raft_consensus.cc:3054] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.106148 2208 raft_consensus.cc:513] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.106719 2208 leader_election.cc:290] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:40.107456 710 leader_election.cc:343] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.107699 1164 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.107884 1164 ts_tablet_manager.cc:1397] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20250112 20:51:40.108942 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829e9dd5854961b5402b5710624612" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.109174 912 raft_consensus.cc:3054] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.112079 912 raft_consensus.cc:2463] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.112511 710 leader_election.cc:304] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.112995 2326 raft_consensus.cc:2798] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.113164 2326 raft_consensus.cc:695] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.113440 2326 consensus_queue.cc:237] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.119720 1185 ts_tablet_manager.cc:1918] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.124985 2355 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:40.125870 1184 ts_tablet_manager.cc:1918] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:40.126669 1185 log.cc:1198] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1a04c223889f424e831e82a52a84f804
I20250112 20:51:40.129192 1185 ts_tablet_manager.cc:1939] T 1a04c223889f424e831e82a52a84f804 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.133602 2371 raft_consensus.cc:491] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.108714 1164 raft_consensus.cc:357] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.134117 1164 raft_consensus.cc:383] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.134112 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:40.134261 1164 raft_consensus.cc:738] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.134480 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:40.134534 1164 consensus_queue.cc:260] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.135282 1164 ts_tablet_manager.cc:1428] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250112 20:51:40.135569 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_d9d198758a4a4d4f83b2a03af392409a [id=1de4457277634138946b0e0710d31cc7] alter complete (version 1)
I20250112 20:51:40.135627 1164 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.137336 1164 tablet_bootstrap.cc:654] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.137563 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_df950a6830fa45b1a8f21feda53ea0ea [id=05211d8527144710860dc2aca1ac76a3] alter complete (version 1)
I20250112 20:51:40.139390 1185 tablet_replica.cc:331] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.139595 1185 raft_consensus.cc:2238] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.139869 1185 pending_rounds.cc:62] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:40.139943 1185 pending_rounds.cc:69] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585046511411200 op_type: NO_OP noop_request { }
I20250112 20:51:40.140084 1185 raft_consensus.cc:2883] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:40.140197 1185 raft_consensus.cc:2267] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.133761 2371 raft_consensus.cc:513] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.144903 1184 log.cc:1198] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7ab5ecd2698b4952a892135b3593106a
I20250112 20:51:40.145310 1184 ts_tablet_manager.cc:1939] T 7ab5ecd2698b4952a892135b3593106a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:40.145627 754 tablet_service.cc:1467] Processing CreateTablet for tablet 16b41160f80c49c0bac6f6922a0ee156 (DEFAULT_TABLE table=default.Table_f962e711e3ff42a69fa1d2ac1b9ec27d [id=95b1cdd3c2584562a2e07555e15e75b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.145781 1185 ts_tablet_manager.cc:1905] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.145870 2371 leader_election.cc:290] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.146018 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b41160f80c49c0bac6f6922a0ee156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.147351 1178 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.147487 1178 ts_tablet_manager.cc:1397] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.169s user 0.002s sys 0.014s
I20250112 20:51:40.148240 1178 raft_consensus.cc:357] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.148496 1178 raft_consensus.cc:383] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.148556 1178 raft_consensus.cc:738] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.148854 1178 consensus_queue.cc:260] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.153558 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 (DEFAULT_TABLE table=default.Table_b9a0b9c1271a440f95be03d7aff157ef [id=6f5fb451f32e4f6f88e12ec15ea038ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.153736 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 (DEFAULT_TABLE table=default.Table_8aba8733d71c40e690d75bc533ebddb3 [id=1184a7ee142c4102b7e2620d18f6ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.153944 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1edc79f17f4bb7a6bb9ab619403142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.154143 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5065a94bb13c419e9d9b8eaca72f5052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.154104 737 tablet_service.cc:1467] Processing CreateTablet for tablet 18daa381181549228ad521157bf8793f (DEFAULT_TABLE table=default.Table_53510fe9a4124c20880d08d01510903c [id=501c7dd39b04491abb764800e9458afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.157649 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd (DEFAULT_TABLE table=default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93 [id=667763bbabf14caf8a4e4cbc079abdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.157980 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2440e0bdf8ad4ab7805c892e28834acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.158277 748 tablet_service.cc:1467] Processing CreateTablet for tablet 38979539a3ae4d08a398874bf894ac32 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.158574 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38979539a3ae4d08a398874bf894ac32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.160995 753 tablet_service.cc:1467] Processing CreateTablet for tablet 61ded646c84746fcb1a71e80996380d8 (DEFAULT_TABLE table=default.Table_c076efa279bf4b74bcb85f1f0d355bed [id=a1eca2669e9d41eeaf1e6987ca84d29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.161307 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ded646c84746fcb1a71e80996380d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.164304 1178 ts_tablet_manager.cc:1428] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 20:51:40.164620 1178 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:40.166882 1185 ts_tablet_manager.cc:1918] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.170100 1186 ts_tablet_manager.cc:1918] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.171819 1186 log.cc:1198] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:40.172245 1186 ts_tablet_manager.cc:1939] T 81b932b7e1e34d4ab18aae40047dc038 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:40.173722 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1a04c223889f424e831e82a52a84f804 (table default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]) successfully deleted
I20250112 20:51:40.171702 1178 tablet_bootstrap.cc:654] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.175077 1186 tablet_replica.cc:331] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.175361 1186 raft_consensus.cc:2238] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.175804 1186 raft_consensus.cc:2267] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.176851 1186 ts_tablet_manager.cc:1905] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.171895 753 tablet_service.cc:1467] Processing CreateTablet for tablet 29809c91e33a491a8503d7951135b15d (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.178138 1185 log.cc:1198] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:40.178416 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29809c91e33a491a8503d7951135b15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.178468 748 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfbcab367040b88dd76ac37e0791c6 (DEFAULT_TABLE table=default.Table_875e19c996d5437894e99baaa6fcffbf [id=6e10783263d0422399bc479c76d0ef56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.178686 1185 ts_tablet_manager.cc:1939] T 4301a07fcb044fd2a6f3e5ef424b411f P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.173137 754 tablet_service.cc:1467] Processing CreateTablet for tablet 056ed0fd01824bcea53d58251328898c (DEFAULT_TABLE table=default.Table_8fae6a2fd2e041f6953157f01d134e04 [id=d4e1323ba12841e980331a881b4afa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.173447 741 tablet_service.cc:1467] Processing CreateTablet for tablet 05e171717b7d462399b141aed9be3c08 (DEFAULT_TABLE table=default.Table_857cd22bec1a484eb38f055272a9a9a8 [id=a9b201c8d37e43469fc225267597b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.177824 756 tablet_service.cc:1467] Processing CreateTablet for tablet dd62b97f353e437dab548c679b28e446 (DEFAULT_TABLE table=default.Table_b4de1d4b39114ec2b50c35a8a02318f9 [id=8bbb306fbb244742b5709ad9c827316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.178134 747 tablet_service.cc:1467] Processing CreateTablet for tablet b29fe38d2fb34ede920f5c497b585110 (DEFAULT_TABLE table=default.Table_4372511dc98c437da09ce4a0aab012f7 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.178865 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cfbcab367040b88dd76ac37e0791c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.180306 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e171717b7d462399b141aed9be3c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.180636 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd62b97f353e437dab548c679b28e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.184876 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056ed0fd01824bcea53d58251328898c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.185114 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29fe38d2fb34ede920f5c497b585110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.185670 1178 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.185810 1178 ts_tablet_manager.cc:1397] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250112 20:51:40.172283 744 tablet_service.cc:1467] Processing CreateTablet for tablet 370ad081bf784808b2ab44cad8d80959 (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.189203 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370ad081bf784808b2ab44cad8d80959. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.191610 1185 tablet_replica.cc:331] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.191788 1185 raft_consensus.cc:2238] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.191926 1185 raft_consensus.cc:2267] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.192410 1185 ts_tablet_manager.cc:1905] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.192498 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4301a07fcb044fd2a6f3e5ef424b411f (table default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]) successfully deleted
I20250112 20:51:40.172582 749 tablet_service.cc:1467] Processing CreateTablet for tablet e0bbf979d0b841899a2990a39d83394e (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.193024 1044 raft_consensus.cc:3054] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.193152 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bbf979d0b841899a2990a39d83394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.196741 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829e9dd5854961b5402b5710624612" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.197062 2355 tablet.cc:1721] T 99ba8529a287434b9ba78e0b6fa7817e P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:40.197499 2208 consensus_queue.cc:1035] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:40.198671 710 leader_election.cc:343] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.200325 2140 raft_consensus.cc:491] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.200487 2140 raft_consensus.cc:513] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.201296 2140 leader_election.cc:290] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.201769 1186 ts_tablet_manager.cc:1918] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.201894 1186 log.cc:1198] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3f0a808d431a43fb9299fec1dcf6d3b9
I20250112 20:51:40.202248 1186 ts_tablet_manager.cc:1939] T 3f0a808d431a43fb9299fec1dcf6d3b9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:40.202466 983 leader_election.cc:343] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:40.202976 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7f79b3309e45afb9e18cad97ddfeb0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:40.203405 983 leader_election.cc:343] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:40.203497 983 leader_election.cc:304] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:40.203896 2140 raft_consensus.cc:2743] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:40.204248 1186 tablet_replica.cc:331] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.204466 1186 raft_consensus.cc:2238] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.204721 1186 raft_consensus.cc:2267] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.205112 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829e9dd5854961b5402b5710624612" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.205276 1186 ts_tablet_manager.cc:1905] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.212426 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_e2d706e7614440a1905f8790f025ac29 [id=75ce02402d924e7f8153b6348a26af9e]) successfully deleted
I20250112 20:51:40.212474 1178 raft_consensus.cc:357] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.212656 1178 raft_consensus.cc:383] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.212702 1178 raft_consensus.cc:738] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.212898 1178 consensus_queue.cc:260] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.220594 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18daa381181549228ad521157bf8793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.225210 1178 ts_tablet_manager.cc:1428] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250112 20:51:40.228447 1186 ts_tablet_manager.cc:1918] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.228639 1186 log.cc:1198] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:40.229111 1186 ts_tablet_manager.cc:1939] T 4301a07fcb044fd2a6f3e5ef424b411f P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:40.248270 2355 tablet.cc:1721] T f5aca1ff0fd541bfa96bf7f279fa4e2d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:40.255139 2371 raft_consensus.cc:491] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.255283 2371 raft_consensus.cc:513] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.255894 2371 leader_election.cc:290] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.261539 1042 raft_consensus.cc:3054] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:40.263468 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 902fc636a0494462ae1e73b0cefc5dfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902fc636a0494462ae1e73b0cefc5dfe already in progress: creating tablet
I20250112 20:51:40.264531 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4f31cf8f24e35b7281d4a8b5667a0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:40.264689 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 60eefdbabc174cd6a67fd12ea7d95071 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60eefdbabc174cd6a67fd12ea7d95071 already in progress: creating tablet
W20250112 20:51:40.265039 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 18664fc1f93f4d48a75223c80da29b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18664fc1f93f4d48a75223c80da29b7a already in progress: creating tablet
I20250112 20:51:40.265482 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4bd46a422093495ba8b5b03039874a32 (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
I20250112 20:51:40.293393 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.295660 1178 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:40.297295 1178 tablet_bootstrap.cc:654] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.297298 2355 tablet.cc:1721] T 38cdb9a415f745cbb62f27763e490d5b P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:40.298121 1186 tablet_replica.cc:331] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.298336 1186 raft_consensus.cc:2238] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.298512 1186 raft_consensus.cc:2267] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.299011 1186 ts_tablet_manager.cc:1905] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.301699 1163 tablet_bootstrap.cc:492] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.301836 1163 ts_tablet_manager.cc:1397] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.245s user 0.002s sys 0.002s
I20250112 20:51:40.302742 1163 raft_consensus.cc:357] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.303006 1163 raft_consensus.cc:383] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.303066 1163 raft_consensus.cc:738] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.303310 1163 consensus_queue.cc:260] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.303925 1163 ts_tablet_manager.cc:1428] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:40.304289 1163 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.306006 1163 tablet_bootstrap.cc:654] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.268193 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c53abd31a1464c22abcb8d42a39cefac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.307337 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c53abd31a1464c22abcb8d42a39cefac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.284298 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_75837bdca38e4a358cb5c44071b13b26" } modify_external_catalogs: true
I20250112 20:51:40.308481 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_75837bdca38e4a358cb5c44071b13b26" } modify_external_catalogs: true
I20250112 20:51:40.300068 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_17b597f08a264f22b57f12049ca81079" } modify_external_catalogs: true
I20250112 20:51:40.308877 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_17b597f08a264f22b57f12049ca81079" } modify_external_catalogs: true
W20250112 20:51:40.310988 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 45525.7us
I20250112 20:51:40.311256 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 81b932b7e1e34d4ab18aae40047dc038 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
W20250112 20:51:40.311546 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cbc22207e9204b73877c74f61ebba8a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbc22207e9204b73877c74f61ebba8a8 already in progress: creating tablet
I20250112 20:51:40.311779 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3f0a808d431a43fb9299fec1dcf6d3b9 (table default.Table_cf3ea3bf6db14c1a8e6bdb8b565e6ad9 [id=3bd13ed71a8e4b23930c7e402367b6d0]) successfully deleted
I20250112 20:51:40.317721 1186 ts_tablet_manager.cc:1918] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:40.318830 1186 log.cc:1198] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7ab5ecd2698b4952a892135b3593106a
I20250112 20:51:40.319649 1186 ts_tablet_manager.cc:1939] T 7ab5ecd2698b4952a892135b3593106a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:40.327873 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5f0cbbb33c64a8bb7de60f2d1ee88e9 consensus state has no leader
I20250112 20:51:40.329088 1186 tablet_replica.cc:331] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.329473 893 tablet_service.cc:1467] Processing CreateTablet for tablet 16b41160f80c49c0bac6f6922a0ee156 (DEFAULT_TABLE table=default.Table_f962e711e3ff42a69fa1d2ac1b9ec27d [id=95b1cdd3c2584562a2e07555e15e75b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.329689 1186 raft_consensus.cc:2238] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.329900 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b41160f80c49c0bac6f6922a0ee156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.330291 1186 raft_consensus.cc:2267] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.331175 1186 ts_tablet_manager.cc:1905] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.336580 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_326b5443cb3245d0bafc2e553c0bfda8" } new_table_name: "default.Table_7927297ff7264be5a6391d2a58f38280" modify_external_catalogs: true
I20250112 20:51:40.340049 1178 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.340214 1178 ts_tablet_manager.cc:1397] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.002s
I20250112 20:51:40.341214 1178 raft_consensus.cc:357] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.341459 1178 raft_consensus.cc:383] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.341521 1178 raft_consensus.cc:738] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.341739 1178 consensus_queue.cc:260] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.346745 1178 ts_tablet_manager.cc:1428] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250112 20:51:40.347113 1178 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:40.350970 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 (DEFAULT_TABLE table=default.Table_b9a0b9c1271a440f95be03d7aff157ef [id=6f5fb451f32e4f6f88e12ec15ea038ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.351286 883 tablet_service.cc:1467] Processing CreateTablet for tablet 370ad081bf784808b2ab44cad8d80959 (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.351481 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 (DEFAULT_TABLE table=default.Table_8aba8733d71c40e690d75bc533ebddb3 [id=1184a7ee142c4102b7e2620d18f6ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.351934 886 tablet_service.cc:1467] Processing CreateTablet for tablet dd62b97f353e437dab548c679b28e446 (DEFAULT_TABLE table=default.Table_b4de1d4b39114ec2b50c35a8a02318f9 [id=8bbb306fbb244742b5709ad9c827316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.352002 884 tablet_service.cc:1467] Processing CreateTablet for tablet b29fe38d2fb34ede920f5c497b585110 (DEFAULT_TABLE table=default.Table_4372511dc98c437da09ce4a0aab012f7 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.351935 874 tablet_service.cc:1467] Processing CreateTablet for tablet 18daa381181549228ad521157bf8793f (DEFAULT_TABLE table=default.Table_53510fe9a4124c20880d08d01510903c [id=501c7dd39b04491abb764800e9458afd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:40.352301 847 leader_election.cc:343] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.352344 880 tablet_service.cc:1467] Processing CreateTablet for tablet 61ded646c84746fcb1a71e80996380d8 (DEFAULT_TABLE table=default.Table_c076efa279bf4b74bcb85f1f0d355bed [id=a1eca2669e9d41eeaf1e6987ca84d29d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:40.352449 847 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.352547 847 leader_election.cc:343] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.352797 882 tablet_service.cc:1467] Processing CreateTablet for tablet 38979539a3ae4d08a398874bf894ac32 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.353281 879 tablet_service.cc:1467] Processing CreateTablet for tablet 05e171717b7d462399b141aed9be3c08 (DEFAULT_TABLE table=default.Table_857cd22bec1a484eb38f055272a9a9a8 [id=a9b201c8d37e43469fc225267597b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.353669 889 tablet_service.cc:1467] Processing CreateTablet for tablet 29809c91e33a491a8503d7951135b15d (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.354007 893 tablet_service.cc:1467] Processing CreateTablet for tablet 056ed0fd01824bcea53d58251328898c (DEFAULT_TABLE table=default.Table_8fae6a2fd2e041f6953157f01d134e04 [id=d4e1323ba12841e980331a881b4afa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.354094 888 tablet_service.cc:1467] Processing CreateTablet for tablet e0bbf979d0b841899a2990a39d83394e (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.354485 877 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfbcab367040b88dd76ac37e0791c6 (DEFAULT_TABLE table=default.Table_875e19c996d5437894e99baaa6fcffbf [id=6e10783263d0422399bc479c76d0ef56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.352128 878 tablet_service.cc:1467] Processing CreateTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd (DEFAULT_TABLE table=default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93 [id=667763bbabf14caf8a4e4cbc079abdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.359063 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1edc79f17f4bb7a6bb9ab619403142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.361765 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5c98d71957e443d48bf632f3fe662c5c" } new_table_name: "default.Table_d3a871e1d7194bfdb60a286d2fe92c1c" modify_external_catalogs: true
I20250112 20:51:40.362207 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370ad081bf784808b2ab44cad8d80959. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.365027 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5065a94bb13c419e9d9b8eaca72f5052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.369004 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd62b97f353e437dab548c679b28e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.369875 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 26c3f3095603490f8861b1b26d5fe78d
I20250112 20:51:40.226286 1185 ts_tablet_manager.cc:1918] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.247179 821 heartbeater.cc:346] Connected to a master server at 127.0.104.60:45153
I20250112 20:51:40.371444 1185 log.cc:1198] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:40.371747 1185 ts_tablet_manager.cc:1939] T 3b7f79b3309e45afb9e18cad97ddfeb0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.261704 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4f31cf8f24e35b7281d4a8b5667a0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.372085 774 raft_consensus.cc:2463] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:40.372140 1185 tablet_replica.cc:331] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.262075 2326 raft_consensus.cc:491] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.265960 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.299221 2208 raft_consensus.cc:491] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.372303 1185 raft_consensus.cc:2238] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.372402 775 raft_consensus.cc:2463] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:40.372380 2208 raft_consensus.cc:513] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.372584 1185 raft_consensus.cc:2267] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.343673 2059 consensus_queue.cc:1035] T e912d588104f482498c40239c9e585b9 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.373003 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29fe38d2fb34ede920f5c497b585110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.373119 1185 ts_tablet_manager.cc:1905] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.373203 2208 leader_election.cc:290] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.354380 1164 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.371153 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c3f3095603490f8861b1b26d5fe78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.373328 2326 raft_consensus.cc:513] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.373437 1164 ts_tablet_manager.cc:1397] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.238s user 0.026s sys 0.017s
I20250112 20:51:40.374169 1164 raft_consensus.cc:357] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.374327 1164 raft_consensus.cc:383] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.374372 1164 raft_consensus.cc:738] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.374542 1164 consensus_queue.cc:260] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.374997 1164 ts_tablet_manager.cc:1428] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 20:51:40.375241 1164 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.375751 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
I20250112 20:51:40.379573 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18daa381181549228ad521157bf8793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.379980 2326 leader_election.cc:290] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:40.381870 1164 tablet_bootstrap.cc:654] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.383479 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ded646c84746fcb1a71e80996380d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.389204 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38979539a3ae4d08a398874bf894ac32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.389611 1185 ts_tablet_manager.cc:1918] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:40.389705 1185 log.cc:1198] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7ab5ecd2698b4952a892135b3593106a
I20250112 20:51:40.390027 1185 ts_tablet_manager.cc:1939] T 7ab5ecd2698b4952a892135b3593106a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.393000 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e171717b7d462399b141aed9be3c08. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:40.393961 847 leader_election.cc:343] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.394563 847 leader_election.cc:343] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.395076 847 leader_election.cc:304] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:40.395248 847 raft_consensus.cc:2681] T 4bd46a422093495ba8b5b03039874a32 P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:40.395805 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29809c91e33a491a8503d7951135b15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.396510 847 leader_election.cc:304] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.396697 2108 raft_consensus.cc:2798] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.396842 2108 raft_consensus.cc:695] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:40.397043 2108 consensus_queue.cc:237] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.398919 2197 raft_consensus.cc:491] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.399065 2197 raft_consensus.cc:513] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.399784 2197 leader_election.cc:290] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.402297 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_00baf585f7a14dd3a9c7998b5956c6e1" } modify_external_catalogs: true
I20250112 20:51:40.402566 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_00baf585f7a14dd3a9c7998b5956c6e1" } modify_external_catalogs: true
I20250112 20:51:40.406823 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ebb3c30194d9fa6a092d9bec4eadf" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.407303 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dcd1a120684446bbe19303fb8416ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.407527 910 raft_consensus.cc:2463] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:40.409662 1163 tablet_bootstrap.cc:492] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.409955 1163 ts_tablet_manager.cc:1397] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.000s
I20250112 20:51:40.410912 1163 raft_consensus.cc:357] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:40.415045 710 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.415300 710 leader_election.cc:304] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.415661 2208 raft_consensus.cc:2798] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.415753 2208 raft_consensus.cc:491] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.415823 2208 raft_consensus.cc:3054] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.417546 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3eeb085e764941198ae7f653bc24757d
I20250112 20:51:40.418268 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 903027fe49e34a308108356ba89a9c3a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:40.418387 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.421247 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cdcbd0285dd04edebfa92654c16e5d39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.421976 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdcbd0285dd04edebfa92654c16e5d39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:40.441211 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 44703.2us
I20250112 20:51:40.444870 1163 raft_consensus.cc:383] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.451403 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_285cfdca273f457c98ac4d491dfe84af [id=e8f4d732342e43bda26b51c8e1122d80] alter complete (version 1)
I20250112 20:51:40.453727 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fc795f0197f74154a56a81c67c43185c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.454330 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc795f0197f74154a56a81c67c43185c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.457075 1163 raft_consensus.cc:738] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.458462 1163 consensus_queue.cc:260] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.455601 2371 raft_consensus.cc:491] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.459048 847 leader_election.cc:304] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.458985 2371 raft_consensus.cc:513] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.459653 2371 leader_election.cc:290] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.459790 2371 raft_consensus.cc:2798] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.459861 1164 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.459888 2371 raft_consensus.cc:695] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:40.459998 1164 ts_tablet_manager.cc:1397] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.085s user 0.019s sys 0.003s
I20250112 20:51:40.460096 2371 consensus_queue.cc:237] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.460704 1164 raft_consensus.cc:357] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.461169 1164 raft_consensus.cc:383] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.461236 1164 raft_consensus.cc:738] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.461444 1164 consensus_queue.cc:260] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.462003 1164 ts_tablet_manager.cc:1428] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:40.462265 1164 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.463774 1164 tablet_bootstrap.cc:654] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.468317 1163 ts_tablet_manager.cc:1428] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.058s user 0.001s sys 0.000s
I20250112 20:51:40.468619 1163 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.470115 1163 tablet_bootstrap.cc:654] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.479274 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bbf979d0b841899a2990a39d83394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.481668 1164 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.481813 1164 ts_tablet_manager.cc:1397] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250112 20:51:40.482605 1164 raft_consensus.cc:357] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.482795 1164 raft_consensus.cc:383] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.482848 1164 raft_consensus.cc:738] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.485091 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cfbcab367040b88dd76ac37e0791c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.489073 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2440e0bdf8ad4ab7805c892e28834acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.497706 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_800fbaf988064a7e8810c2617a6c8863 [id=1ef2730aa400453a9d078c520a5329f8] alter complete (version 1)
I20250112 20:51:40.499498 1164 consensus_queue.cc:260] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.500361 1164 ts_tablet_manager.cc:1428] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20250112 20:51:40.500674 1164 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.502036 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9856e6b625d04f17b0869cee08f69142
I20250112 20:51:40.504215 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056ed0fd01824bcea53d58251328898c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.505597 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.505795 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.505945 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.506105 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.506242 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:40.509227 453 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d0829e9dd5854961b5402b5710624612 consensus state has no leader
W20250112 20:51:40.509900 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 47) took 857 ms (client timeout 1000 ms). Trace:
I20250112 20:51:40.511699 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 725e56fa8ac343e5b1ec1214da02b7d4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:40.512876 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.513679 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.513854 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.514071 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.514199 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.514304 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.514557 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:40.510031 453 rpcz_store.cc:269] 0112 20:51:39.652889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:39.652933 (+ 44us) service_pool.cc:225] Handling call
0112 20:51:40.509866 (+856933us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:39.841414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:39.841530 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:39.841536 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:39.841539 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:39.841681 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:39.841689 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:39.841694 (+ 5us) tablet.cc:682] Acquiring locks for 3 operations
0112 20:51:39.841785 (+ 91us) tablet.cc:698] Row locks acquired
0112 20:51:39.841789 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:39.841823 (+ 34us) write_op.cc:270] Start()
0112 20:51:39.841852 (+ 29us) write_op.cc:276] Timestamp: P: 1736715099841709 usec, L: 0
0112 20:51:39.841856 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:39.842026 (+ 170us) log.cc:844] Serialized 1275 byte log entry
0112 20:51:39.913595 (+ 71569us) op_driver.cc:464] REPLICATION: finished
0112 20:51:39.914425 (+ 830us) write_op.cc:301] APPLY: starting
0112 20:51:39.914435 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:39.914473 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:39.914504 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:39.914578 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:39.914604 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 20:51:39.914611 (+ 7us) write_op.cc:312] APPLY: finished
0112 20:51:39.914664 (+ 53us) log.cc:844] Serialized 35 byte log entry
0112 20:51:39.914729 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:39.914743 (+ 14us) write_op.cc:454] Released schema lock
0112 20:51:39.914756 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:39.914816 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:39.914819 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 20:51:40.091353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:40.091480 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:40.091486 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:40.091490 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:40.091584 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:40.091596 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:40.091604 (+ 8us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:40.091660 (+ 56us) tablet.cc:698] Row locks acquired
0112 20:51:40.091663 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:40.091692 (+ 29us) write_op.cc:270] Start()
0112 20:51:40.091719 (+ 27us) write_op.cc:276] Timestamp: P: 1736715100091672 usec, L: 0
0112 20:51:40.091723 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:40.091895 (+ 172us) log.cc:844] Serialized 356 byte log entry
0112 20:51:40.110656 (+ 18761us) op_driver.cc:464] REPLICATION: finished
0112 20:51:40.111370 (+ 714us) write_op.cc:301] APPLY: starting
0112 20:51:40.111388 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:40.111429 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:40.111461 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:40.111513 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:40.111544 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:40.111552 (+ 8us) write_op.cc:312] APPLY: finished
0112 20:51:40.111621 (+ 69us) log.cc:844] Serialized 23 byte log entry
0112 20:51:40.111854 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:40.111873 (+ 19us) write_op.cc:454] Released schema lock
0112 20:51:40.111881 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:40.122831 (+ 10950us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:40.122837 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 20:51:40.455873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:40.455971 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:40.455976 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:40.455978 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:40.456060 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:40.456070 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:40.456074 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:40.456124 (+ 50us) tablet.cc:698] Row locks acquired
0112 20:51:40.456126 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:40.456156 (+ 30us) write_op.cc:270] Start()
0112 20:51:40.456185 (+ 29us) write_op.cc:276] Timestamp: P: 1736715100456099 usec, L: 0
0112 20:51:40.456189 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:40.456354 (+ 165us) log.cc:844] Serialized 356 byte log entry
0112 20:51:40.495774 (+ 39420us) op_driver.cc:464] REPLICATION: finished
0112 20:51:40.496295 (+ 521us) write_op.cc:301] APPLY: starting
0112 20:51:40.496312 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:40.496345 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:40.496377 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:40.496424 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:40.496452 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:40.496461 (+ 9us) write_op.cc:312] APPLY: finished
0112 20:51:40.496527 (+ 66us) log.cc:844] Serialized 23 byte log entry
0112 20:51:40.496586 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:40.496601 (+ 15us) write_op.cc:454] Released schema lock
0112 20:51:40.496607 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:40.496692 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:40.496697 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":792,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":3,"prepare.queue_time_us":72281,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":986,"replication_time_us":71713}],["op",{"apply.queue_time_us":657,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":649,"num_ops":1,"prepare.queue_time_us":169899,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":31459,"raft.queue_time_us":117,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":635,"replication_time_us":18901}],["op",{"apply.queue_time_us":466,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":377,"num_ops":1,"prepare.queue_time_us":317144,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":15768,"raft.queue_time_us":51,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":272,"replication_time_us":39546}]]}
I20250112 20:51:40.516942 1178 tablet_bootstrap.cc:700] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Time spent opening tablet: real 0.169s user 0.002s sys 0.000s
I20250112 20:51:40.517283 1178 tablet_bootstrap.cc:654] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.526345 1185 tablet_replica.cc:331] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.526556 1185 raft_consensus.cc:2238] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.526746 1185 raft_consensus.cc:2267] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.527303 1185 ts_tablet_manager.cc:1905] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.531602 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_e2d706e7614440a1905f8790f025ac29 [id=75ce02402d924e7f8153b6348a26af9e]) successfully deleted
W20250112 20:51:40.531885 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b532068d2374b59a3c46525db7c50db because the tablet was not found. No further retry: Not found: Tablet not found: 3b532068d2374b59a3c46525db7c50db
W20250112 20:51:40.532150 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 26c3f3095603490f8861b1b26d5fe78d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c3f3095603490f8861b1b26d5fe78d already in progress: creating tablet
I20250112 20:51:40.502264 1164 tablet_bootstrap.cc:654] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:40.549917 2406 log.cc:926] Time spent T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: Append to log took a long time: real 0.077s user 0.004s sys 0.000s
I20250112 20:51:40.553258 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55640 (request call id 0) took 2471 ms. Trace:
I20250112 20:51:40.553591 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ae76622ac46f5a584a582e372bc05. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:40.554009 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 20:51:40.557056 2384 raft_consensus.cc:491] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.557076 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_18294fc7a4b448b2a1f57904a112b19d" } new_table_name: "default.Table_fbe3a476695f40ec8737a0e3e4ad731f" modify_external_catalogs: true
I20250112 20:51:40.557243 2384 raft_consensus.cc:513] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.558048 2384 leader_election.cc:290] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:40.559383 983 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b310e4135c2b424fb006a38a60e41066
W20250112 20:51:40.559902 983 leader_election.cc:343] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: b310e4135c2b424fb006a38a60e41066
I20250112 20:51:40.559996 983 leader_election.cc:304] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:40.560427 2384 raft_consensus.cc:2743] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:40.561336 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1de9e41db6aa48deb9875ff6f40a87bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.561985 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de9e41db6aa48deb9875ff6f40a87bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:40.563334 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 103874us
I20250112 20:51:40.564144 1164 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.564308 1164 ts_tablet_manager.cc:1397] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250112 20:51:40.565142 1164 raft_consensus.cc:357] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.565335 1164 raft_consensus.cc:383] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.565441 1164 raft_consensus.cc:738] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.565706 1164 consensus_queue.cc:260] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.566381 1164 ts_tablet_manager.cc:1428] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:40.566653 1164 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.568187 1164 tablet_bootstrap.cc:654] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.569939 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 49a9e6cf325d48cc9a2a956400a68deb
I20250112 20:51:40.570688 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a53e0c8faffe4744926220db2ddb69f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.571233 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a53e0c8faffe4744926220db2ddb69f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.572762 913 raft_consensus.cc:1270] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3 [term 2 FOLLOWER]: Refusing update from remote peer 30ba399c7a6749e0acd075d93e4cc4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 20:51:40.573354 2384 consensus_queue.cc:1035] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.582877 2379 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:40.583102 2379 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:40.584913 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.585908 2379 leader_election.cc:290] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271)
I20250112 20:51:40.587726 847 leader_election.cc:304] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.588393 847 leader_election.cc:304] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.589395 2108 raft_consensus.cc:2798] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.589665 2415 raft_consensus.cc:2798] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.589711 2108 raft_consensus.cc:491] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.589936 2108 raft_consensus.cc:3054] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.589816 2415 raft_consensus.cc:491] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.590265 2415 raft_consensus.cc:3054] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.593770 1163 tablet_bootstrap.cc:492] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.594493 2208 raft_consensus.cc:513] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.595285 2208 leader_election.cc:290] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:40.595706 1186 ts_tablet_manager.cc:1918] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.595819 1186 log.cc:1198] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1a04c223889f424e831e82a52a84f804
I20250112 20:51:40.595880 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dcd1a120684446bbe19303fb8416ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.596132 1186 ts_tablet_manager.cc:1939] T 1a04c223889f424e831e82a52a84f804 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:40.596135 912 raft_consensus.cc:3054] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.601011 2108 raft_consensus.cc:513] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.601859 2108 leader_election.cc:290] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.602265 1163 ts_tablet_manager.cc:1397] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.000s
I20250112 20:51:40.603212 1163 raft_consensus.cc:357] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.603585 1163 raft_consensus.cc:383] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.603670 1163 raft_consensus.cc:738] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.603901 1163 consensus_queue.cc:260] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.604503 1163 ts_tablet_manager.cc:1428] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:40.604847 1163 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.607884 1164 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.608035 1164 ts_tablet_manager.cc:1397] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20250112 20:51:40.611552 1164 raft_consensus.cc:357] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.611752 1164 raft_consensus.cc:383] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.611819 1164 raft_consensus.cc:738] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.613058 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4301a07fcb044fd2a6f3e5ef424b411f (table default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]) successfully deleted
I20250112 20:51:40.614713 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7ab5ecd2698b4952a892135b3593106a (table default.Table_e2d706e7614440a1905f8790f025ac29 [id=75ce02402d924e7f8153b6348a26af9e]) successfully deleted
I20250112 20:51:40.616508 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ff8ac6f4d94c10b03a764bbf910c44" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.616695 774 raft_consensus.cc:2463] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:40.618108 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ff8ac6f4d94c10b03a764bbf910c44" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.617515 1164 consensus_queue.cc:260] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.620657 1164 ts_tablet_manager.cc:1428] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 20:51:40.621860 1164 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.623498 1164 tablet_bootstrap.cc:654] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.624430 2090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" instance_seqno: 1736715082556960) as {username='slave'} at 127.0.104.1:38341; Asking this server to re-register.
I20250112 20:51:40.628641 2208 raft_consensus.cc:491] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.628760 2208 raft_consensus.cc:513] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.629324 2208 leader_election.cc:290] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.631448 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2469127891524be5ac3dc13fbcf35c7a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.631640 910 raft_consensus.cc:2463] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:40.631997 710 leader_election.cc:304] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.632292 821 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:40.634186 912 raft_consensus.cc:2463] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.632293 2208 raft_consensus.cc:2798] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.637336 2208 raft_consensus.cc:491] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.637424 2208 raft_consensus.cc:3054] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.637459 2090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" instance_seqno: 1736715083039354) as {username='slave'} at 127.0.104.2:51263; Asking this server to re-register.
I20250112 20:51:40.638429 1185 ts_tablet_manager.cc:1918] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.638528 1185 log.cc:1198] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3eeb085e764941198ae7f653bc24757d
I20250112 20:51:40.638828 1185 ts_tablet_manager.cc:1939] T 3eeb085e764941198ae7f653bc24757d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.639954 2208 raft_consensus.cc:513] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.640532 2208 leader_election.cc:290] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.633729 2324 raft_consensus.cc:491] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.634575 710 leader_election.cc:304] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.641294 1178 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.641418 1178 ts_tablet_manager.cc:1397] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.294s user 0.017s sys 0.012s
I20250112 20:51:40.641379 2324 raft_consensus.cc:513] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:40.641955 2324 leader_election.cc:290] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:40.642082 2324 raft_consensus.cc:2798] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.642169 2324 raft_consensus.cc:695] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.642379 2324 consensus_queue.cc:237] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:40.648310 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0bbf979d0b841899a2990a39d83394e consensus state has no leader
W20250112 20:51:40.649724 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be1f12369c894682b7d30de4e37bd466 consensus state has no leader
I20250112 20:51:40.650339 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8c6a19ee7a4e4afab12870e48ee8df7f
I20250112 20:51:40.651432 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 43d341f6ac6848d9a8e60f689dd24a66 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:40.652041 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:40.642083 2415 raft_consensus.cc:513] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.644065 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570daec845f40778c48ff3f2e2e30cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.644202 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2469127891524be5ac3dc13fbcf35c7a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.653723 912 raft_consensus.cc:3054] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:40.654114 710 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.654728 2415 leader_election.cc:290] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.657012 1178 raft_consensus.cc:357] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.658950 1178 raft_consensus.cc:383] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.657403 912 raft_consensus.cc:2463] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.659071 1178 raft_consensus.cc:738] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.659992 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 16b41160f80c49c0bac6f6922a0ee156 (DEFAULT_TABLE table=default.Table_f962e711e3ff42a69fa1d2ac1b9ec27d [id=95b1cdd3c2584562a2e07555e15e75b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.660244 1178 consensus_queue.cc:260] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.660444 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b41160f80c49c0bac6f6922a0ee156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.660817 1178 ts_tablet_manager.cc:1428] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:40.661090 1178 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:40.662027 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eeb085e764941198ae7f653bc24757d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.663448 1178 tablet_bootstrap.cc:654] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.665086 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 056ed0fd01824bcea53d58251328898c (DEFAULT_TABLE table=default.Table_8fae6a2fd2e041f6953157f01d134e04 [id=d4e1323ba12841e980331a881b4afa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.665426 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056ed0fd01824bcea53d58251328898c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.666065 1163 tablet_bootstrap.cc:654] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.668886 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4f31cf8f24e35b7281d4a8b5667a0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.670032 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c3f3095603490f8861b1b26d5fe78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.669934 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_632fa444ce9749b6a40c85532cb2e6c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.670462 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_632fa444ce9749b6a40c85532cb2e6c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.670754 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.670953 1186 tablet_replica.cc:331] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.671108 1186 raft_consensus.cc:2238] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.671231 1186 raft_consensus.cc:2267] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.671696 1186 ts_tablet_manager.cc:1905] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.671983 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8399a4497e68459fb8e495749b55f0a3" } new_table_name: "default.Table_745bd9e4da3f419aa5aef5b16e9e9d58" modify_external_catalogs: true
I20250112 20:51:40.673445 775 raft_consensus.cc:1270] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Refusing update from remote peer 30ba399c7a6749e0acd075d93e4cc4b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250112 20:51:40.673576 775 pending_rounds.cc:77] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Aborting all ops after (but not including) 0
I20250112 20:51:40.673621 775 pending_rounds.cc:99] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250112 20:51:40.673717 710 leader_election.cc:304] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.673965 2059 raft_consensus.cc:2798] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.674067 2059 raft_consensus.cc:695] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.673488 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4f31cf8f24e35b7281d4a8b5667a0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:40.674191 770 raft_consensus.cc:3054] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.674304 2059 consensus_queue.cc:237] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.673707 775 raft_consensus.cc:2883] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.676723 770 raft_consensus.cc:2463] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:40.677736 1185 tablet_replica.cc:331] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.677928 1185 raft_consensus.cc:2238] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.678054 1185 raft_consensus.cc:2267] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.677979 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eeb085e764941198ae7f653bc24757d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:40.678516 1185 ts_tablet_manager.cc:1905] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.678541 2421 catalog_manager.cc:1260] Loaded cluster ID: 55c7c6228b4d4e3e80a4f1e922af635d
I20250112 20:51:40.678608 2421 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: loading cluster ID for follower catalog manager: success
I20250112 20:51:40.678977 2384 consensus_queue.cc:1035] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.680840 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3eeb085e764941198ae7f653bc24757d (table default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]) successfully deleted
W20250112 20:51:40.681110 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 because the tablet was not found. No further retry: Not found: Tablet not found: 3f0a808d431a43fb9299fec1dcf6d3b9
W20250112 20:51:40.681298 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 because the tablet was not found. No further retry: Not found: Tablet not found: 3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:40.685595 1185 ts_tablet_manager.cc:1918] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.688380 1186 ts_tablet_manager.cc:1918] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.688539 1186 log.cc:1198] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3eeb085e764941198ae7f653bc24757d
I20250112 20:51:40.688968 1186 ts_tablet_manager.cc:1939] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:40.687198 2208 consensus_queue.cc:1035] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.693001 2326 consensus_queue.cc:1035] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.695060 1185 log.cc:1198] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:40.695413 1185 ts_tablet_manager.cc:1939] T 5d384628ba9b4f81b768eda4f796632b P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.696442 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5d384628ba9b4f81b768eda4f796632b (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
I20250112 20:51:40.696779 1185 tablet_replica.cc:331] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.697016 1185 raft_consensus.cc:2238] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.697249 1185 pending_rounds.cc:62] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:40.697335 1185 pending_rounds.cc:69] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585049478643712 op_type: NO_OP noop_request { }
I20250112 20:51:40.697479 1185 raft_consensus.cc:2883] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:40.697618 1185 raft_consensus.cc:2267] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.699070 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8dd9ede7c9b34004b6ccc0e2e9043284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.699579 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dd9ede7c9b34004b6ccc0e2e9043284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.699937 1163 tablet_bootstrap.cc:492] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.700100 1163 ts_tablet_manager.cc:1397] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.000s
I20250112 20:51:40.700786 1163 raft_consensus.cc:357] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.701050 1163 raft_consensus.cc:383] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.701108 1163 raft_consensus.cc:738] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.701295 1163 consensus_queue.cc:260] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.701725 1163 ts_tablet_manager.cc:1428] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:40.702046 1163 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.703648 1163 tablet_bootstrap.cc:654] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.705174 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.705509 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.706712 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.707127 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.707633 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.708001 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.713627 2421 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: acquiring CA information for follower catalog manager: success
I20250112 20:51:40.716104 1186 tablet_replica.cc:331] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:40.716732 1186 raft_consensus.cc:2238] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.717310 1186 pending_rounds.cc:62] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:40.717682 1186 pending_rounds.cc:69] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585051228528640 op_type: NO_OP noop_request { }
I20250112 20:51:40.718001 1186 raft_consensus.cc:2883] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:40.720865 1186 raft_consensus.cc:2267] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.725240 1186 ts_tablet_manager.cc:1905] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.726099 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 (DEFAULT_TABLE table=default.Table_b9a0b9c1271a440f95be03d7aff157ef [id=6f5fb451f32e4f6f88e12ec15ea038ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.726482 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b1edc79f17f4bb7a6bb9ab619403142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.732609 2208 raft_consensus.cc:491] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.732741 2208 raft_consensus.cc:513] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.733049 2421 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 20:51:40.733402 2208 leader_election.cc:290] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.735270 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 (DEFAULT_TABLE table=default.Table_8aba8733d71c40e690d75bc533ebddb3 [id=1184a7ee142c4102b7e2620d18f6ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.735785 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5065a94bb13c419e9d9b8eaca72f5052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.736135 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ceb775b58bf04d71a6e9cec1b517610d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.736752 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceb775b58bf04d71a6e9cec1b517610d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.738960 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f863419259044f8799ad66ba8d35c47e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.739578 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f863419259044f8799ad66ba8d35c47e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.740658 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4372511dc98c437da09ce4a0aab012f7" } new_table_name: "default.Table_4be3bb2de6134efd8f12da08aa470d25" modify_external_catalogs: true
I20250112 20:51:40.726224 1178 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.741334 1178 ts_tablet_manager.cc:1397] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.008s
I20250112 20:51:40.742029 1178 raft_consensus.cc:357] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.742237 1178 raft_consensus.cc:383] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.742290 1178 raft_consensus.cc:738] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.734464 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbb679585fc44699553fe85ea044566" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.742473 1178 consensus_queue.cc:260] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.742988 1178 ts_tablet_manager.cc:1428] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:40.743225 1178 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:40.742828 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e35d3834d4b64573afe5a5ab0cac708b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.743868 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e35d3834d4b64573afe5a5ab0cac708b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.744446 1178 tablet_bootstrap.cc:654] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.746742 912 raft_consensus.cc:2463] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:40.747391 710 leader_election.cc:304] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.747901 2208 raft_consensus.cc:2798] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.748013 2208 raft_consensus.cc:491] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.748114 2208 raft_consensus.cc:3054] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.750725 2208 raft_consensus.cc:513] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.751974 2208 leader_election.cc:290] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.752255 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbb679585fc44699553fe85ea044566" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.752461 912 raft_consensus.cc:3054] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.758668 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.759168 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 18daa381181549228ad521157bf8793f (DEFAULT_TABLE table=default.Table_53510fe9a4124c20880d08d01510903c [id=501c7dd39b04491abb764800e9458afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.759603 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18daa381181549228ad521157bf8793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.761662 958 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:40.764351 1185 ts_tablet_manager.cc:1905] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.773545 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.775101 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd (DEFAULT_TABLE table=default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93 [id=667763bbabf14caf8a4e4cbc079abdca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.775642 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2440e0bdf8ad4ab7805c892e28834acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.785902 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.786741 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 61ded646c84746fcb1a71e80996380d8 (DEFAULT_TABLE table=default.Table_c076efa279bf4b74bcb85f1f0d355bed [id=a1eca2669e9d41eeaf1e6987ca84d29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.787197 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ded646c84746fcb1a71e80996380d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.788749 1164 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:40.788900 1164 ts_tablet_manager.cc:1397] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.167s user 0.005s sys 0.013s
I20250112 20:51:40.789610 1164 raft_consensus.cc:357] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.789798 1164 raft_consensus.cc:383] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.789845 1164 raft_consensus.cc:738] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.790030 1164 consensus_queue.cc:260] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.790503 1164 ts_tablet_manager.cc:1428] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:40.790665 2090 ts_manager.cc:194] Registered new tserver with Master: aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.790722 1164 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:40.791419 1185 ts_tablet_manager.cc:1918] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.791522 1185 log.cc:1198] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4bd46a422093495ba8b5b03039874a32
I20250112 20:51:40.791800 2443 raft_consensus.cc:491] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.791893 1185 ts_tablet_manager.cc:1939] T 4bd46a422093495ba8b5b03039874a32 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.791939 2443 raft_consensus.cc:513] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.792995 2443 leader_election.cc:290] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:40.794550 983 leader_election.cc:343] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:40.794999 1185 tablet_replica.cc:331] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.795264 1185 raft_consensus.cc:2238] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.795513 1185 pending_rounds.cc:62] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:40.795600 1185 pending_rounds.cc:69] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585049984057344 op_type: NO_OP noop_request { }
I20250112 20:51:40.795737 1185 raft_consensus.cc:2883] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:40.795877 1185 raft_consensus.cc:2267] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.796607 1185 ts_tablet_manager.cc:1905] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.796875 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4bd46a422093495ba8b5b03039874a32 (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
W20250112 20:51:40.797250 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9856e6b625d04f17b0869cee08f69142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9856e6b625d04f17b0869cee08f69142 already in progress: creating tablet
W20250112 20:51:40.802098 983 leader_election.cc:343] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:40.802232 983 leader_election.cc:304] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:40.802732 2443 raft_consensus.cc:2743] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:40.804669 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:40.808388 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:40.808926 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 38979539a3ae4d08a398874bf894ac32 (DEFAULT_TABLE table=default.Table_03812fa225b44a6ab342958dd68ec4c2 [id=e4dd7b3ee5fc489a853f9081ee127be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.809486 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38979539a3ae4d08a398874bf894ac32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.810400 2059 raft_consensus.cc:491] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.810526 2059 raft_consensus.cc:513] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.811053 2059 leader_election.cc:290] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.813583 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 05e171717b7d462399b141aed9be3c08 (DEFAULT_TABLE table=default.Table_857cd22bec1a484eb38f055272a9a9a8 [id=a9b201c8d37e43469fc225267597b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.813935 1164 tablet_bootstrap.cc:654] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.813956 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e171717b7d462399b141aed9be3c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.816222 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36935e6d8c5643388ad163458e2b387c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.816432 910 raft_consensus.cc:2463] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:40.816835 710 leader_election.cc:304] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.817083 2059 raft_consensus.cc:2798] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.817147 2059 raft_consensus.cc:491] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.817202 2059 raft_consensus.cc:3054] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.819651 2059 raft_consensus.cc:513] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.820814 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36935e6d8c5643388ad163458e2b387c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:40.821004 913 raft_consensus.cc:3054] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.823848 913 raft_consensus.cc:2463] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.824330 710 leader_election.cc:304] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.824594 2059 leader_election.cc:290] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.824762 2059 raft_consensus.cc:2798] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.824978 2059 raft_consensus.cc:695] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.825327 2059 consensus_queue.cc:237] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.828641 1185 ts_tablet_manager.cc:1918] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.828752 1185 log.cc:1198] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:40.829171 1185 ts_tablet_manager.cc:1939] T 9c6d5d425e6a493daa46900cbd8db166 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.832753 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1a04c223889f424e831e82a52a84f804 (table default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]) successfully deleted
I20250112 20:51:40.833896 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:40.833900 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3facda2ee004b2e9487194ba040c9cf" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.834084 774 raft_consensus.cc:3054] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.834221 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 29809c91e33a491a8503d7951135b15d (DEFAULT_TABLE table=default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.834741 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29809c91e33a491a8503d7951135b15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.841778 1347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:40.842708 2032 leader_election.cc:304] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5c09e899184de6b770dca873501062; no voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:40.842840 774 raft_consensus.cc:2463] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:40.843175 2379 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:40.844436 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9c6d5d425e6a493daa46900cbd8db166 (table default.Table_a9245c5997d7416fa75ebf6202856645 [id=942d944d07b146fbb1e30e593fa0a266]) successfully deleted
I20250112 20:51:40.847481 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 370ad081bf784808b2ab44cad8d80959 (DEFAULT_TABLE table=default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.847887 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370ad081bf784808b2ab44cad8d80959. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:40.848457 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1a04c223889f424e831e82a52a84f804 because the tablet was not found. No further retry: Not found: Tablet not found: 1a04c223889f424e831e82a52a84f804
I20250112 20:51:40.849061 1185 tablet_replica.cc:331] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.850180 1185 raft_consensus.cc:2238] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.853319 1185 raft_consensus.cc:2267] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.854583 1185 ts_tablet_manager.cc:1905] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:40.857343 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 26c3f3095603490f8861b1b26d5fe78d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c3f3095603490f8861b1b26d5fe78d already in progress: creating tablet
I20250112 20:51:40.858340 912 raft_consensus.cc:2463] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:40.859141 1015 tablet_service.cc:1467] Processing CreateTablet for tablet e0bbf979d0b841899a2990a39d83394e (DEFAULT_TABLE table=default.Table_326b5443cb3245d0bafc2e553c0bfda8 [id=8530d1e179394e18ad4464ae2187dcf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.859545 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0bbf979d0b841899a2990a39d83394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.864894 1012 tablet_service.cc:1467] Processing CreateTablet for tablet dd62b97f353e437dab548c679b28e446 (DEFAULT_TABLE table=default.Table_b4de1d4b39114ec2b50c35a8a02318f9 [id=8bbb306fbb244742b5709ad9c827316b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.865229 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd62b97f353e437dab548c679b28e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.868744 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b29fe38d2fb34ede920f5c497b585110 (DEFAULT_TABLE table=default.Table_4372511dc98c437da09ce4a0aab012f7 [id=e2a5927a7809438fa42a6fb940952e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.869092 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29fe38d2fb34ede920f5c497b585110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.872293 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c3f3095603490f8861b1b26d5fe78d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.872551 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.872835 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.873008 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a04c223889f424e831e82a52a84f804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.873176 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7f79b3309e45afb9e18cad97ddfeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:28 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:40.873183 710 leader_election.cc:343] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.873502 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6a19ee7a4e4afab12870e48ee8df7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.872844 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a9e6cf325d48cc9a2a956400a68deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:40.874948 2090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" instance_seqno: 1736715083518275) as {username='slave'} at 127.0.104.3:46365; Asking this server to re-register.
I20250112 20:51:40.876108 2208 raft_consensus.cc:491] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.876303 2208 raft_consensus.cc:513] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.877799 2208 leader_election.cc:290] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.881337 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3eeb085e764941198ae7f653bc24757d (table default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]) successfully deleted
W20250112 20:51:40.883637 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9856e6b625d04f17b0869cee08f69142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9856e6b625d04f17b0869cee08f69142 already in progress: creating tablet
W20250112 20:51:40.884037 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 81b932b7e1e34d4ab18aae40047dc038 because the tablet was not found. No further retry: Not found: Tablet not found: 81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:40.899338 2443 raft_consensus.cc:491] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:40.899477 2443 raft_consensus.cc:513] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.900089 2443 leader_election.cc:290] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.900673 2447 tablet_replica.cc:331] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:40.901294 2447 raft_consensus.cc:2238] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:40.901540 2447 raft_consensus.cc:2267] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.902087 710 leader_election.cc:304] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:40.878374 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:40.902725 2447 ts_tablet_manager.cc:1905] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:40.902740 710 leader_election.cc:343] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.903481 2440 raft_consensus.cc:2798] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:40.903532 983 leader_election.cc:343] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:40.903620 2440 raft_consensus.cc:695] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:40.903836 2440 consensus_queue.cc:237] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:40.904191 983 leader_election.cc:343] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:40.904294 983 leader_election.cc:304] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:40.904752 2443 raft_consensus.cc:2743] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:40.905493 2090 ts_manager.cc:194] Registered new tserver with Master: 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:40.905838 1185 ts_tablet_manager.cc:1918] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.906069 2324 consensus_queue.cc:1035] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.907732 1185 log.cc:1198] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:40.908185 1185 ts_tablet_manager.cc:1939] T 944b71897bcf4304bd9f7bb7adef34e5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.913287 1010 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfbcab367040b88dd76ac37e0791c6 (DEFAULT_TABLE table=default.Table_875e19c996d5437894e99baaa6fcffbf [id=6e10783263d0422399bc479c76d0ef56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.913453 1185 tablet_replica.cc:331] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.913635 1185 raft_consensus.cc:2238] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:40.913662 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cfbcab367040b88dd76ac37e0791c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.913688 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 944b71897bcf4304bd9f7bb7adef34e5 (table default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]) successfully deleted
I20250112 20:51:40.913897 1185 raft_consensus.cc:2267] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.915011 1185 ts_tablet_manager.cc:1905] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.918748 1186 ts_tablet_manager.cc:1918] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.920603 1185 ts_tablet_manager.cc:1918] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.925774 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ebb3c30194d9fa6a092d9bec4eadf" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.931113 2447 ts_tablet_manager.cc:1918] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:40.933061 1185 log.cc:1198] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/64887b3578284cb286ee122ab8c626c4
I20250112 20:51:40.933327 2447 log.cc:1198] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1a04c223889f424e831e82a52a84f804
I20250112 20:51:40.934090 1185 ts_tablet_manager.cc:1939] T 64887b3578284cb286ee122ab8c626c4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.934451 2447 ts_tablet_manager.cc:1939] T 1a04c223889f424e831e82a52a84f804 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:40.935635 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 64887b3578284cb286ee122ab8c626c4 (table default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]) successfully deleted
W20250112 20:51:40.941123 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b310e4135c2b424fb006a38a60e41066 because the tablet was not found. No further retry: Not found: Tablet not found: b310e4135c2b424fb006a38a60e41066
I20250112 20:51:40.941298 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a" } modify_external_catalogs: true
I20250112 20:51:40.941474 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a" } modify_external_catalogs: true
I20250112 20:51:40.942255 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_07581b86150545baa24899a6b2452aee" } modify_external_catalogs: true
W20250112 20:51:40.942354 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 81b932b7e1e34d4ab18aae40047dc038 because the tablet was not found. No further retry: Not found: Tablet not found: 81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:40.942441 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_07581b86150545baa24899a6b2452aee" } modify_external_catalogs: true
W20250112 20:51:40.943156 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4301a07fcb044fd2a6f3e5ef424b411f because the tablet was not found. No further retry: Not found: Tablet not found: 4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:40.943414 1185 tablet_replica.cc:331] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:40.943603 1185 raft_consensus.cc:2238] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:40.943708 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b29fe38d2fb34ede920f5c497b585110 consensus state has no leader
I20250112 20:51:40.943759 1185 raft_consensus.cc:2267] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:40.943964 451 master_service.cc:946] ReplaceTablet: received request to replace tablet d0829e9dd5854961b5402b5710624612 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:40.944487 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f6995f7fbbf480995353b66749fbc10 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:40.945147 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c53e401a48884786b8e51ae3d66e8d87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:40.946228 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c53e401a48884786b8e51ae3d66e8d87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:40.946283 1185 ts_tablet_manager.cc:1905] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:40.954635 1163 tablet_bootstrap.cc:492] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:40.954777 1163 ts_tablet_manager.cc:1397] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.253s user 0.002s sys 0.000s
I20250112 20:51:40.955739 1163 raft_consensus.cc:357] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.955928 1163 raft_consensus.cc:383] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.955968 1163 raft_consensus.cc:738] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.956125 1163 consensus_queue.cc:260] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:40.958175 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet be1f12369c894682b7d30de4e37bd466 (table default.Table_d3a871e1d7194bfdb60a286d2fe92c1c [id=3a57e9e997904845bd32bfdc2ae17d88], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet be1f12369c894682b7d30de4e37bd466 consensus state has no leader
I20250112 20:51:40.958755 449 catalog_manager.cc:5526] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.959529 449 catalog_manager.cc:5526] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.959904 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:40.960728 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dcd1a120684446bbe19303fb8416ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.960170 449 catalog_manager.cc:5526] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.961949 449 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]). Expected version 1 got 0
I20250112 20:51:40.962424 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dcd1a120684446bbe19303fb8416ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.962622 1185 ts_tablet_manager.cc:1918] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:40.963246 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2469127891524be5ac3dc13fbcf35c7a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.963312 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_a56ab4e8789f432ba82943e7830afea7 [id=69e87fbf3a29404d92d4b2902996d147] alter complete (version 1)
I20250112 20:51:40.963610 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2469127891524be5ac3dc13fbcf35c7a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.963944 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570daec845f40778c48ff3f2e2e30cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.964609 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbb679585fc44699553fe85ea044566" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.964747 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5d48f415092e41a3ada61770f9908c9d" } modify_external_catalogs: true
I20250112 20:51:40.964985 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5d48f415092e41a3ada61770f9908c9d" } modify_external_catalogs: true
I20250112 20:51:40.965008 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbb679585fc44699553fe85ea044566" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.965909 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36935e6d8c5643388ad163458e2b387c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.966284 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36935e6d8c5643388ad163458e2b387c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:40.966630 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:40.967041 1041 raft_consensus.cc:3054] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:40.967571 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cf1dc64f1964202bfa8169388a34781 consensus state has no leader
I20250112 20:51:40.968276 1185 log.cc:1198] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/49a9e6cf325d48cc9a2a956400a68deb
I20250112 20:51:40.968623 1185 ts_tablet_manager.cc:1939] T 49a9e6cf325d48cc9a2a956400a68deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:40.969599 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 49a9e6cf325d48cc9a2a956400a68deb (table default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]) successfully deleted
W20250112 20:51:40.970675 710 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.970809 710 leader_election.cc:304] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:40.971362 2326 raft_consensus.cc:2743] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:40.973845 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 26c3f3095603490f8861b1b26d5fe78d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c3f3095603490f8861b1b26d5fe78d already in progress: creating tablet
W20250112 20:51:40.974090 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9856e6b625d04f17b0869cee08f69142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9856e6b625d04f17b0869cee08f69142 already in progress: creating tablet
W20250112 20:51:40.974313 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3eeb085e764941198ae7f653bc24757d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3eeb085e764941198ae7f653bc24757d already in progress: creating tablet
I20250112 20:51:40.974915 737 tablet_service.cc:1467] Processing CreateTablet for tablet 8103d90b5c764ead9780e37a8160c4f7 (DEFAULT_TABLE table=default.Table_8ac2217609124d6ba7936c9268f10af7 [id=9dc27e0e54ca4e57b2be86266f6bdf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.975281 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8103d90b5c764ead9780e37a8160c4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.976823 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b2a82b2202e4bc6b5164fe9c4f07829" } modify_external_catalogs: true
I20250112 20:51:40.977001 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b2a82b2202e4bc6b5164fe9c4f07829" } modify_external_catalogs: true
I20250112 20:51:40.980177 1163 ts_tablet_manager.cc:1428] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250112 20:51:40.980445 1163 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:40.980940 753 tablet_service.cc:1467] Processing CreateTablet for tablet 734734ff9c564f769bea0984a9e0386b (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.981349 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734734ff9c564f769bea0984a9e0386b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.982129 737 tablet_service.cc:1467] Processing CreateTablet for tablet c058dcab55ca49298ff544dcb8c3f754 (DEFAULT_TABLE table=default.Table_99673c7bb11c4ff5a0b8879e673d3c3d [id=72335492497a49d4893aff0b17cf145b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.982249 741 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd (DEFAULT_TABLE table=default.Table_28f569ded7ee456cb79ce58b8adb809d [id=c3a512398e9b4904a274b0f63bcde097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.982447 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058dcab55ca49298ff544dcb8c3f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.982636 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:40.886263 847 leader_election.cc:343] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.984901 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 98639.6us
I20250112 20:51:40.985033 744 tablet_service.cc:1467] Processing CreateTablet for tablet 77b1978f129c43dcb9e77e76540f7ac4 (DEFAULT_TABLE table=default.Table_10c19a0eede64f589d42d79a287f52f9 [id=f6a1eda9415a4ff981d4eb0fcf757288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.985334 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b1978f129c43dcb9e77e76540f7ac4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:40.985975 847 leader_election.cc:343] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.986486 847 leader_election.cc:343] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:40.986626 847 leader_election.cc:343] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:40.988077 847 leader_election.cc:304] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.988324 2415 raft_consensus.cc:2798] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:40.988396 2415 raft_consensus.cc:491] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.988454 2415 raft_consensus.cc:3054] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.989006 749 tablet_service.cc:1467] Processing CreateTablet for tablet bb5331ad9c544514be691d1c1cd10b9a (DEFAULT_TABLE table=default.Table_d587a303a8b44cecacc313b337eb84b1 [id=54f91fd054d944089745ae46e6102947]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.989317 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb5331ad9c544514be691d1c1cd10b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.991042 2415 raft_consensus.cc:513] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.991602 2415 leader_election.cc:290] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:40.993054 747 tablet_service.cc:1467] Processing CreateTablet for tablet d339bc4c8cd1481ebcfd721512976691 (DEFAULT_TABLE table=default.Table_45f57bdcc8a24f6695242226c7422b8c [id=2ca8265e8ae946f887c485f2bffff3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.993360 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d339bc4c8cd1481ebcfd721512976691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:40.993963 847 leader_election.cc:304] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:40.994210 2415 raft_consensus.cc:2798] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.994415 2415 raft_consensus.cc:695] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:40.994637 2415 consensus_queue.cc:237] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:40.997146 756 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac551767ed4e1c8da08e0e1177a7f2 (DEFAULT_TABLE table=default.Table_b9e4f2b60aef4766b60f2655988f3a57 [id=ef6dbfab1286406f81d97103dcb570e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:40.940538 1178 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:40.997457 1178 ts_tablet_manager.cc:1397] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.254s user 0.022s sys 0.039s
I20250112 20:51:40.998083 1178 raft_consensus.cc:357] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.998261 1178 raft_consensus.cc:383] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.998306 1178 raft_consensus.cc:738] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.998467 1178 consensus_queue.cc:260] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:40.998934 1178 ts_tablet_manager.cc:1428] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:40.999181 1178 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.000509 1178 tablet_bootstrap.cc:654] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.003636 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ac551767ed4e1c8da08e0e1177a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.016009 1164 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:41.017225 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c6a19ee7a4e4afab12870e48ee8df7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6a19ee7a4e4afab12870e48ee8df7f already in progress: creating tablet
I20250112 20:51:41.017632 1164 ts_tablet_manager.cc:1397] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.227s user 0.007s sys 0.013s
I20250112 20:51:41.018623 1164 raft_consensus.cc:357] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.023191 1164 raft_consensus.cc:383] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.023432 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 61b31b8f4d2e4eea84849e3f935cf46e
I20250112 20:51:41.023432 1164 raft_consensus.cc:738] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.024258 1164 consensus_queue.cc:260] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.024809 1164 ts_tablet_manager.cc:1428] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:41.025902 2447 tablet_replica.cc:331] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.026090 2447 raft_consensus.cc:2238] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.026219 2447 raft_consensus.cc:2267] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.026677 2447 ts_tablet_manager.cc:1905] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.028533 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c5469ad8d235490ebfbbb933c659d6ec" } new_table_name: "default.Table_ee4fb486face46d4801238c334cdfce8" modify_external_catalogs: true
I20250112 20:51:41.029495 1178 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.029620 1178 ts_tablet_manager.cc:1397] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20250112 20:51:41.029914 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 43d341f6ac6848d9a8e60f689dd24a66
I20250112 20:51:41.030400 455 catalog_manager.cc:6596] ReplaceTablet: tablet 43d341f6ac6848d9a8e60f689dd24a66 deleted and replaced by tablet 7d9572179a6f45759214c6c39f22b2f0
I20250112 20:51:41.030750 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cc39ddff5b22412fb084d77879afe4da" } new_table_name: "default.Table_9b3f8eb107694ac5a7c7de6d5609352f" modify_external_catalogs: true
I20250112 20:51:41.030995 1178 raft_consensus.cc:357] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.031180 1178 raft_consensus.cc:383] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.031226 1178 raft_consensus.cc:738] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.031422 1178 consensus_queue.cc:260] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.031900 1178 ts_tablet_manager.cc:1428] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:41.031936 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e2c41c2c92b442b986d6b75708c7420c
I20250112 20:51:41.032554 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_830c03785e1a464fb83fcb19bcd333e3" } modify_external_catalogs: true
I20250112 20:51:41.032802 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c41c2c92b442b986d6b75708c7420c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.032948 1178 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.032951 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.033011 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8f6995f7fbbf480995353b66749fbc10
I20250112 20:51:41.033473 447 catalog_manager.cc:6596] ReplaceTablet: tablet 8f6995f7fbbf480995353b66749fbc10 deleted and replaced by tablet 443b29a93a0245e9a3a256e65a272699
I20250112 20:51:41.033649 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f6995f7fbbf480995353b66749fbc10 with delete_type TABLET_DATA_DELETED (replaced by tablet 443b29a93a0245e9a3a256e65a272699) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.033828 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.034302 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.034749 2440 raft_consensus.cc:491] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.034873 2440 raft_consensus.cc:513] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.035005 1178 tablet_bootstrap.cc:654] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.035871 1164 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.032723 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_830c03785e1a464fb83fcb19bcd333e3" } modify_external_catalogs: true
I20250112 20:51:41.037500 1164 tablet_bootstrap.cc:654] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:41.040870 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e2c41c2c92b442b986d6b75708c7420c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c41c2c92b442b986d6b75708c7420c already in progress: creating tablet
W20250112 20:51:41.043066 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 43d341f6ac6848d9a8e60f689dd24a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43d341f6ac6848d9a8e60f689dd24a66 already in progress: creating tablet
W20250112 20:51:41.043743 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8f6995f7fbbf480995353b66749fbc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f6995f7fbbf480995353b66749fbc10 already in progress: creating tablet
I20250112 20:51:41.047833 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f34aa32d6474561988b2cf3d0e8fd3b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:41.048484 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1f1dd19c189f4b1aa0faab3dac476a4a
I20250112 20:51:41.049286 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2ee684deb6634f31981eab910abbc0fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.049788 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ee684deb6634f31981eab910abbc0fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.052459 710 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.053573 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:41.054065 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1f1dd19c189f4b1aa0faab3dac476a4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1dd19c189f4b1aa0faab3dac476a4a already in progress: creating tablet
I20250112 20:51:41.055805 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.056784 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.056839 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f52a9216c5044f88a0060e3e8e95eddd
I20250112 20:51:41.056977 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.057137 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.057273 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.057407 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.057439 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2ac80cd3fa134256b29e1eb39796aa6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:41.057555 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.057936 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f52a9216c5044f88a0060e3e8e95eddd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:41.057981 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53551931b23d43da89e9193cdf034a63 consensus state has no leader
I20250112 20:51:41.058087 753 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.058326 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a113b0bc0435433eb5260a68d20896f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.058794 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a113b0bc0435433eb5260a68d20896f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.060292 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:41.060516 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:41.062059 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.062501 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.064186 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.064522 1185 tablet_replica.cc:331] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.064716 1185 raft_consensus.cc:2238] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.064968 1185 pending_rounds.cc:62] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:41.065016 1185 pending_rounds.cc:69] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585052233568256 op_type: NO_OP noop_request { }
I20250112 20:51:41.065117 1185 raft_consensus.cc:2883] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.065420 1185 raft_consensus.cc:2267] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.066066 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.067056 2326 consensus_queue.cc:1035] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:41.067953 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d587a303a8b44cecacc313b337eb84b1" } new_table_name: "default.Table_d85e56502a8649b19478d46868000abc" modify_external_catalogs: true
W20250112 20:51:41.069753 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ac80cd3fa134256b29e1eb39796aa6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.070891 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8074b00eda8d434faaff4b98ae923ebe consensus state has no leader
I20250112 20:51:41.072463 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]). Expected version 1 got 0
I20250112 20:51:41.075517 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9a23281d9b6f47a5bf627374fa0cfbf2
I20250112 20:51:41.081794 1185 ts_tablet_manager.cc:1905] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.085199 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.094367 2326 consensus_queue.cc:1035] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:41.095468 1185 ts_tablet_manager.cc:1918] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.095583 1185 log.cc:1198] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:41.095927 1185 ts_tablet_manager.cc:1939] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.097087 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 35dcd1a120684446bbe19303fb8416ef (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
I20250112 20:51:41.097720 2447 ts_tablet_manager.cc:1918] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.097918 2447 log.cc:1198] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3b7f79b3309e45afb9e18cad97ddfeb0
I20250112 20:51:41.101193 1185 tablet_replica.cc:331] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.101464 1185 raft_consensus.cc:2238] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:41.102417 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 64887b3578284cb286ee122ab8c626c4 because the tablet was not found. No further retry: Not found: Tablet not found: 64887b3578284cb286ee122ab8c626c4
I20250112 20:51:41.108359 1185 raft_consensus.cc:2267] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.117738 1185 ts_tablet_manager.cc:1905] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.120051 2440 leader_election.cc:290] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.123988 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d0829e9dd5854961b5402b5710624612
I20250112 20:51:41.124469 451 catalog_manager.cc:6596] ReplaceTablet: tablet d0829e9dd5854961b5402b5710624612 deleted and replaced by tablet 7623bebb539e485aa508e95a35225979
I20250112 20:51:41.125712 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829e9dd5854961b5402b5710624612 with delete_type TABLET_DATA_DELETED (replaced by tablet 7623bebb539e485aa508e95a35225979) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:41.130944 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb5331ad9c544514be691d1c1cd10b9a consensus state has no leader
I20250112 20:51:41.141057 1178 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.141208 1178 ts_tablet_manager.cc:1397] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.007s
I20250112 20:51:41.141894 1178 raft_consensus.cc:357] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.142098 1178 raft_consensus.cc:383] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.142148 1178 raft_consensus.cc:738] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.142324 1178 consensus_queue.cc:260] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.142820 1178 ts_tablet_manager.cc:1428] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:41.143110 1178 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.143973 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:41.144484 1178 tablet_bootstrap.cc:654] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:41.144532 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.033382 847 leader_election.cc:343] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250112 20:51:41.145481 847 leader_election.cc:304] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:41.145646 847 raft_consensus.cc:2681] T 3eeb085e764941198ae7f653bc24757d P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:41.145874 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5df2936b4476476fa09325d619af0a27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.146191 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 112809us
W20250112 20:51:41.146418 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5df2936b4476476fa09325d619af0a27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.146503 847 leader_election.cc:304] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:41.146719 2415 raft_consensus.cc:2798] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:41.146848 2415 raft_consensus.cc:695] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:41.147078 2415 consensus_queue.cc:237] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.147526 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_99673c7bb11c4ff5a0b8879e673d3c3d" } new_table_name: "default.Table_d97953504d9648629174944c086c84ef" modify_external_catalogs: true
W20250112 20:51:41.148226 710 leader_election.cc:343] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.148458 1164 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.148566 1164 ts_tablet_manager.cc:1397] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.004s
I20250112 20:51:41.148674 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 4301a07fcb044fd2a6f3e5ef424b411f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:41.148713 710 leader_election.cc:343] T 35dcd1a120684446bbe19303fb8416ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.148811 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5a2776514bca4b24ade2fab5a7ed71df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:41.149260 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b932b7e1e34d4ab18aae40047dc038 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fd34e829d0645ea80daaff5d27cb266) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:41.149317 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a2776514bca4b24ade2fab5a7ed71df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.149446 710 consensus_peers.cc:487] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.149695 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 8103d90b5c764ead9780e37a8160c4f7 (DEFAULT_TABLE table=default.Table_8ac2217609124d6ba7936c9268f10af7 [id=9dc27e0e54ca4e57b2be86266f6bdf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.150040 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8103d90b5c764ead9780e37a8160c4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.150686 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_40709325c312462982486fcd141ade99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.151335 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40709325c312462982486fcd141ade99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.147089 1186 log.cc:1198] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:41.154544 1186 ts_tablet_manager.cc:1939] T 5d384628ba9b4f81b768eda4f796632b P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.158567 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2527a0aaec5d441f8399cb2d2a6aa813 (DEFAULT_TABLE table=default.Table_f863419259044f8799ad66ba8d35c47e [id=87b4191b598f4360a6cd8ad6fe2de9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.158640 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:41.158818 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 36935e6d8c5643388ad163458e2b387c (table default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51]). Expected version 1 got 0
I20250112 20:51:41.159000 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:41.159085 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2527a0aaec5d441f8399cb2d2a6aa813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.159196 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]). Expected version 1 got 0
I20250112 20:51:41.159685 753 tablet_service.cc:1467] Processing CreateTablet for tablet c39a3858c52a4deabb019b6e2c3d4074 (DEFAULT_TABLE table=default.Table_8dd9ede7c9b34004b6ccc0e2e9043284 [id=5a7c2ba0e01543609d0ac3b612c0f30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.159976 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39a3858c52a4deabb019b6e2c3d4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.161032 737 tablet_service.cc:1467] Processing CreateTablet for tablet 40656387a5fd4c379ff2c6ef663df1a2 (DEFAULT_TABLE table=default.Table_ceb775b58bf04d71a6e9cec1b517610d [id=459ae6f5e4b14457b5bf8aa76114557f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.161353 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40656387a5fd4c379ff2c6ef663df1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.162021 1186 tablet_replica.cc:331] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.162192 1186 raft_consensus.cc:2238] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.162369 1186 pending_rounds.cc:62] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:41.162403 1186 pending_rounds.cc:69] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585053676556288 op_type: NO_OP noop_request { }
I20250112 20:51:41.162503 1186 raft_consensus.cc:2883] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.162560 1186 raft_consensus.cc:2267] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.165971 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]). Expected version 1 got 0
I20250112 20:51:41.166218 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]). Expected version 2 got 0
I20250112 20:51:41.166626 455 catalog_manager.cc:5526] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.167197 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 36935e6d8c5643388ad163458e2b387c (table default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51]). Expected version 1 got 0
I20250112 20:51:41.167404 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]). Expected version 1 got 0
I20250112 20:51:41.167595 455 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 06253bb13a9541d8b408d1bf73e469dd (table default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7]). Expected version 1 got 0
I20250112 20:51:41.169047 744 tablet_service.cc:1467] Processing CreateTablet for tablet d5afc68113174c179d33fac8f46e4de5 (DEFAULT_TABLE table=default.Table_632fa444ce9749b6a40c85532cb2e6c6 [id=df8bf02bb9b44292beebebe400c8667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.169370 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5afc68113174c179d33fac8f46e4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.169610 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ff8ac6f4d94c10b03a764bbf910c44" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:41.170832 1186 ts_tablet_manager.cc:1905] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.173908 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5d384628ba9b4f81b768eda4f796632b (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
I20250112 20:51:41.174687 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8103d90b5c764ead9780e37a8160c4f7 (DEFAULT_TABLE table=default.Table_8ac2217609124d6ba7936c9268f10af7 [id=9dc27e0e54ca4e57b2be86266f6bdf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.175073 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8103d90b5c764ead9780e37a8160c4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.177017 882 tablet_service.cc:1467] Processing CreateTablet for tablet 734734ff9c564f769bea0984a9e0386b (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.177269 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734734ff9c564f769bea0984a9e0386b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.180330 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_861997ae234b444fae5e7160ed43a3e7" } new_table_name: "default.Table_db5f5dc69d334043b179fca5b701f975" modify_external_catalogs: true
I20250112 20:51:41.182897 888 tablet_service.cc:1467] Processing CreateTablet for tablet bb5331ad9c544514be691d1c1cd10b9a (DEFAULT_TABLE table=default.Table_d587a303a8b44cecacc313b337eb84b1 [id=54f91fd054d944089745ae46e6102947]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.183195 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb5331ad9c544514be691d1c1cd10b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.184942 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f6995f7fbbf480995353b66749fbc10 with delete_type TABLET_DATA_DELETED (replaced by tablet 443b29a93a0245e9a3a256e65a272699) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.185180 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.185709 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f52a9216c5044f88a0060e3e8e95eddd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.187868 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 762388f687dd4b05ae6ffdaf8c64c999 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:41.188870 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.189042 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.189167 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.189289 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.189530 889 tablet_service.cc:1467] Processing CreateTablet for tablet c058dcab55ca49298ff544dcb8c3f754 (DEFAULT_TABLE table=default.Table_99673c7bb11c4ff5a0b8879e673d3c3d [id=72335492497a49d4893aff0b17cf145b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.189815 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058dcab55ca49298ff544dcb8c3f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.190623 1164 raft_consensus.cc:357] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.190811 1164 raft_consensus.cc:383] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.190856 1164 raft_consensus.cc:738] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.191068 1164 consensus_queue.cc:260] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.191562 1164 ts_tablet_manager.cc:1428] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.043s user 0.001s sys 0.000s
I20250112 20:51:41.191912 875 tablet_service.cc:1467] Processing CreateTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 (DEFAULT_TABLE table=default.Table_cdcbd0285dd04edebfa92654c16e5d39 [id=db95b74a655c4f57a166e73c8a6ed63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.191998 887 tablet_service.cc:1467] Processing CreateTablet for tablet d5afc68113174c179d33fac8f46e4de5 (DEFAULT_TABLE table=default.Table_632fa444ce9749b6a40c85532cb2e6c6 [id=df8bf02bb9b44292beebebe400c8667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.192229 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b062c1d67d4443eabe8ac1d8a832c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.192346 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.192638 881 tablet_service.cc:1467] Processing CreateTablet for tablet b040a8eefbe44fd49f6cf0bd2fdcf651 (DEFAULT_TABLE table=default.Table_7afee759e98341ce855fc733cb16a5f9 [id=b0880b74fadf49a2a202d0e7602b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.192953 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b040a8eefbe44fd49f6cf0bd2fdcf651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.193022 874 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac551767ed4e1c8da08e0e1177a7f2 (DEFAULT_TABLE table=default.Table_b9e4f2b60aef4766b60f2655988f3a57 [id=ef6dbfab1286406f81d97103dcb570e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.193315 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ac551767ed4e1c8da08e0e1177a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.197001 883 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd (DEFAULT_TABLE table=default.Table_28f569ded7ee456cb79ce58b8adb809d [id=c3a512398e9b4904a274b0f63bcde097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.197031 876 tablet_service.cc:1467] Processing CreateTablet for tablet 36f95046c87846b2aa7e15430c1480b5 (DEFAULT_TABLE table=default.Table_fc795f0197f74154a56a81c67c43185c [id=09f9919ce6db46dfb9a6527e7d7e7a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.197252 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5afc68113174c179d33fac8f46e4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.197345 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.197580 879 tablet_service.cc:1467] Processing CreateTablet for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5 (DEFAULT_TABLE table=default.Table_e35d3834d4b64573afe5a5ab0cac708b [id=77be21069e934b91ab76bce3194366c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.197826 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c41c2c92b442b986d6b75708c7420c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.198062 886 tablet_service.cc:1467] Processing CreateTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 (DEFAULT_TABLE table=default.Table_c53abd31a1464c22abcb8d42a39cefac [id=af47182c306c4cc39ef2a02eba51d73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.198395 1185 ts_tablet_manager.cc:1918] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.198494 1185 log.cc:1198] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:41.198490 892 tablet_service.cc:1467] Processing CreateTablet for tablet beeaf42f79834d4f890aa6690c786562 (DEFAULT_TABLE table=default.Table_ecd375c288164374b6c8e4927fd9bc32 [id=c8bb5600a83d4d1890ddc10dd9f54d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.198762 1185 ts_tablet_manager.cc:1939] T 267ebb3c30194d9fa6a092d9bec4eadf P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.198873 891 tablet_service.cc:1467] Processing CreateTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 (DEFAULT_TABLE table=default.Table_a53e0c8faffe4744926220db2ddb69f9 [id=02b953cc10a3404391c5e29313552312]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.199214 877 tablet_service.cc:1467] Processing CreateTablet for tablet 77b1978f129c43dcb9e77e76540f7ac4 (DEFAULT_TABLE table=default.Table_10c19a0eede64f589d42d79a287f52f9 [id=f6a1eda9415a4ff981d4eb0fcf757288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.199509 884 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829e9dd5854961b5402b5710624612 with delete_type TABLET_DATA_DELETED (replaced by tablet 7623bebb539e485aa508e95a35225979) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.199733 884 tablet_service.cc:1467] Processing CreateTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 (DEFAULT_TABLE table=default.Table_1de9e41db6aa48deb9875ff6f40a87bf [id=ec04d31413a04ffa9f1529189bb6a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.199934 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.200088 878 tablet_service.cc:1467] Processing CreateTablet for tablet d339bc4c8cd1481ebcfd721512976691 (DEFAULT_TABLE table=default.Table_45f57bdcc8a24f6695242226c7422b8c [id=2ca8265e8ae946f887c485f2bffff3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.200361 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2527a0aaec5d441f8399cb2d2a6aa813 (DEFAULT_TABLE table=default.Table_f863419259044f8799ad66ba8d35c47e [id=87b4191b598f4360a6cd8ad6fe2de9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.200574 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.200634 880 tablet_service.cc:1467] Processing CreateTablet for tablet 40656387a5fd4c379ff2c6ef663df1a2 (DEFAULT_TABLE table=default.Table_ceb775b58bf04d71a6e9cec1b517610d [id=459ae6f5e4b14457b5bf8aa76114557f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.200940 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2527a0aaec5d441f8399cb2d2a6aa813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.202013 1164 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.203377 1164 tablet_bootstrap.cc:654] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.204128 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeaf42f79834d4f890aa6690c786562. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:41.205094 847 leader_election.cc:343] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.207175 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f7a47089a43d8ae9c2fc16d379740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.208634 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b1978f129c43dcb9e77e76540f7ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.209160 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.213044 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed925a5ef3a54f819ca292a4e7967b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.215340 747 tablet_service.cc:1467] Processing CreateTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 (DEFAULT_TABLE table=default.Table_a53e0c8faffe4744926220db2ddb69f9 [id=02b953cc10a3404391c5e29313552312]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.215713 741 tablet_service.cc:1467] Processing CreateTablet for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5 (DEFAULT_TABLE table=default.Table_e35d3834d4b64573afe5a5ab0cac708b [id=77be21069e934b91ab76bce3194366c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:41.216153 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c058dcab55ca49298ff544dcb8c3f754 consensus state has no leader
I20250112 20:51:41.216220 748 tablet_service.cc:1467] Processing CreateTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 (DEFAULT_TABLE table=default.Table_1de9e41db6aa48deb9875ff6f40a87bf [id=ec04d31413a04ffa9f1529189bb6a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.216418 754 tablet_service.cc:1467] Processing CreateTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 (DEFAULT_TABLE table=default.Table_c53abd31a1464c22abcb8d42a39cefac [id=af47182c306c4cc39ef2a02eba51d73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.216770 749 tablet_service.cc:1467] Processing CreateTablet for tablet b040a8eefbe44fd49f6cf0bd2fdcf651 (DEFAULT_TABLE table=default.Table_7afee759e98341ce855fc733cb16a5f9 [id=b0880b74fadf49a2a202d0e7602b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.215710 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f7a47089a43d8ae9c2fc16d379740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.217931 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed925a5ef3a54f819ca292a4e7967b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.219550 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.219885 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961a72a9d8a643cea743f3f021bd5a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.220541 737 tablet_service.cc:1467] Processing CreateTablet for tablet beeaf42f79834d4f890aa6690c786562 (DEFAULT_TABLE table=default.Table_ecd375c288164374b6c8e4927fd9bc32 [id=c8bb5600a83d4d1890ddc10dd9f54d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.220875 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b040a8eefbe44fd49f6cf0bd2fdcf651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.221077 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeaf42f79834d4f890aa6690c786562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.224887 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]). Expected version 1 got 0
I20250112 20:51:41.225179 446 catalog_manager.cc:5526] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.225605 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:41.225816 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 9cf1dc64f1964202bfa8169388a34781 (table default.Table_745bd9e4da3f419aa5aef5b16e9e9d58 [id=7e2c24519e17479fb708264c8543222b]). Expected version 1 got 0
W20250112 20:51:41.226775 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0866871b9e1542e8ab30505c68f7dea9 consensus state has no leader
I20250112 20:51:41.227129 744 tablet_service.cc:1467] Processing CreateTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 (DEFAULT_TABLE table=default.Table_cdcbd0285dd04edebfa92654c16e5d39 [id=db95b74a655c4f57a166e73c8a6ed63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.227409 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b062c1d67d4443eabe8ac1d8a832c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.228116 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f95046c87846b2aa7e15430c1480b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.228371 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d339bc4c8cd1481ebcfd721512976691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.228618 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961a72a9d8a643cea743f3f021bd5a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.230403 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40656387a5fd4c379ff2c6ef663df1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.230764 753 tablet_service.cc:1467] Processing CreateTablet for tablet 36f95046c87846b2aa7e15430c1480b5 (DEFAULT_TABLE table=default.Table_fc795f0197f74154a56a81c67c43185c [id=09f9919ce6db46dfb9a6527e7d7e7a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.231086 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f95046c87846b2aa7e15430c1480b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.231345 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0bb556ec81044d1f8296628212fc4875" } modify_external_catalogs: true
I20250112 20:51:41.231508 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0bb556ec81044d1f8296628212fc4875" } modify_external_catalogs: true
I20250112 20:51:41.232014 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e4d7d365450448129def61e210e853ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.232511 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4d7d365450448129def61e210e853ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.233362 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.233691 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.236266 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.238874 1163 tablet_bootstrap.cc:654] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.238904 2447 ts_tablet_manager.cc:1939] T 3b7f79b3309e45afb9e18cad97ddfeb0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:41.239993 2447 tablet_replica.cc:331] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.240226 2447 raft_consensus.cc:2238] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.240386 2447 raft_consensus.cc:2267] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.237846 888 tablet_service.cc:1467] Processing CreateTablet for tablet c39a3858c52a4deabb019b6e2c3d4074 (DEFAULT_TABLE table=default.Table_8dd9ede7c9b34004b6ccc0e2e9043284 [id=5a7c2ba0e01543609d0ac3b612c0f30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.240877 2447 ts_tablet_manager.cc:1905] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.240942 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39a3858c52a4deabb019b6e2c3d4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.242126 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 762388f687dd4b05ae6ffdaf8c64c999
I20250112 20:51:41.242563 448 catalog_manager.cc:6596] ReplaceTablet: tablet 762388f687dd4b05ae6ffdaf8c64c999 deleted and replaced by tablet 2ff5efaf91174cfcbd56e2f23ae08ed0
I20250112 20:51:41.242991 1185 tablet_replica.cc:331] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.246201 1185 raft_consensus.cc:2238] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.246419 1185 pending_rounds.cc:62] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:41.246459 1185 pending_rounds.cc:69] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585053307006976 op_type: NO_OP noop_request { }
I20250112 20:51:41.249514 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 267ebb3c30194d9fa6a092d9bec4eadf (table default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]) successfully deleted
W20250112 20:51:41.249763 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3eeb085e764941198ae7f653bc24757d because the tablet was not found. No further retry: Not found: Tablet not found: 3eeb085e764941198ae7f653bc24757d
I20250112 20:51:41.250319 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c09ffcd01d594a1d8b065295cf69258a" } modify_external_catalogs: true
I20250112 20:51:41.250496 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c09ffcd01d594a1d8b065295cf69258a" } modify_external_catalogs: true
I20250112 20:51:41.246546 1185 raft_consensus.cc:2883] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.252424 1185 raft_consensus.cc:2267] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.254257 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5860b3a08e52455f9708d82fe27361b9" } modify_external_catalogs: true
I20250112 20:51:41.254449 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5860b3a08e52455f9708d82fe27361b9" } modify_external_catalogs: true
I20250112 20:51:41.255838 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8b7b788acf8047acb34ca2596bbf141b
I20250112 20:51:41.259322 2447 ts_tablet_manager.cc:1918] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.259431 2447 log.cc:1198] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/49a9e6cf325d48cc9a2a956400a68deb
I20250112 20:51:41.259775 2447 ts_tablet_manager.cc:1939] T 49a9e6cf325d48cc9a2a956400a68deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:41.260697 2447 tablet_replica.cc:331] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.260907 2447 raft_consensus.cc:2238] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.261052 2447 raft_consensus.cc:2267] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.260737 1185 ts_tablet_manager.cc:1905] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.261492 2447 ts_tablet_manager.cc:1905] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.265257 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ff5efaf91174cfcbd56e2f23ae08ed0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.267155 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b788acf8047acb34ca2596bbf141b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.267160 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1dd4ddaed71542489ede444900ece37b" } new_table_name: "default.Table_3c5be7fd026a4048b1e177fe5956ac0a" modify_external_catalogs: true
I20250112 20:51:41.268409 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 3dba3d9315db4b9292288ee62c2244bc from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:41.269158 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2a71d5cf4d36447da49bad300ee1b392" } modify_external_catalogs: true
I20250112 20:51:41.269402 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2a71d5cf4d36447da49bad300ee1b392" } modify_external_catalogs: true
I20250112 20:51:41.271971 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 77b1978f129c43dcb9e77e76540f7ac4 (DEFAULT_TABLE table=default.Table_10c19a0eede64f589d42d79a287f52f9 [id=f6a1eda9415a4ff981d4eb0fcf757288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.272373 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b1978f129c43dcb9e77e76540f7ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.271971 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 734734ff9c564f769bea0984a9e0386b (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.273105 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734734ff9c564f769bea0984a9e0386b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.276978 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f720bd9710d2476d9632699a2732cb05
I20250112 20:51:41.285498 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f720bd9710d2476d9632699a2732cb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.286381 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:41.286664 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 9cf1dc64f1964202bfa8169388a34781 (table default.Table_745bd9e4da3f419aa5aef5b16e9e9d58 [id=7e2c24519e17479fb708264c8543222b]). Expected version 1 got 0
I20250112 20:51:41.287851 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7f659b016002468f88844b86ed218e32" } modify_external_catalogs: true
I20250112 20:51:41.288034 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7f659b016002468f88844b86ed218e32" } modify_external_catalogs: true
I20250112 20:51:41.288689 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eb719443baaa46e49179af022eae1a4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.289173 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb719443baaa46e49179af022eae1a4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.291241 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f263ea98be854666aeb1b47042a00e85" } new_table_name: "default.Table_e4a5db71ae7a4a31a4648186d20c3324" modify_external_catalogs: true
I20250112 20:51:41.292474 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c79ce3118cbf49e287c93a76398521da (table default.Table_3c5be7fd026a4048b1e177fe5956ac0a [id=c7f63522332840b6bc6cd7462015ad3d]). Expected version 1 got 0
I20250112 20:51:41.293128 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ff5efaf91174cfcbd56e2f23ae08ed0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.293826 1012 tablet_service.cc:1467] Processing CreateTablet for tablet bb5331ad9c544514be691d1c1cd10b9a (DEFAULT_TABLE table=default.Table_d587a303a8b44cecacc313b337eb84b1 [id=54f91fd054d944089745ae46e6102947]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.294291 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb5331ad9c544514be691d1c1cd10b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.300052 875 tablet_service.cc:1514] Processing DeleteTablet for tablet f720bd9710d2476d9632699a2732cb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.301020 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.302270 749 tablet_service.cc:1467] Processing CreateTablet for tablet 5311cd2a18654994a2fdb88be3e9490b (DEFAULT_TABLE table=default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb [id=ed24e86b77b444b095e2f85c3b1994b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.302615 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5311cd2a18654994a2fdb88be3e9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.303296 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:41.303421 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dea98c29bc744009ab9b3eeb53684f29 consensus state has no leader
I20250112 20:51:41.304257 747 tablet_service.cc:1467] Processing CreateTablet for tablet fc1335522a384eb98163a2539630b93e (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.304615 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1335522a384eb98163a2539630b93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.313030 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5808d790defa44d7b6067522207c1022 (DEFAULT_TABLE table=default.Table_2ee684deb6634f31981eab910abbc0fb [id=891c37ab536643a5a84cce4f394c691c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.313445 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5808d790defa44d7b6067522207c1022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.293022 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b788acf8047acb34ca2596bbf141b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.318943 753 tablet_service.cc:1467] Processing CreateTablet for tablet da93a5d375334552bc68510aebb507dd (DEFAULT_TABLE table=default.Table_2ac80cd3fa134256b29e1eb39796aa6d [id=e8c3c2cddc8042c692d8abe7a15aa05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.319275 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93a5d375334552bc68510aebb507dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.323398 889 tablet_service.cc:1467] Processing CreateTablet for tablet da93a5d375334552bc68510aebb507dd (DEFAULT_TABLE table=default.Table_2ac80cd3fa134256b29e1eb39796aa6d [id=e8c3c2cddc8042c692d8abe7a15aa05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.323853 883 tablet_service.cc:1467] Processing CreateTablet for tablet 5808d790defa44d7b6067522207c1022 (DEFAULT_TABLE table=default.Table_2ee684deb6634f31981eab910abbc0fb [id=891c37ab536643a5a84cce4f394c691c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.324231 882 tablet_service.cc:1467] Processing CreateTablet for tablet fc1335522a384eb98163a2539630b93e (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.327329 741 tablet_service.cc:1467] Processing CreateTablet for tablet c23f5f085dd84d68beb277676225e7e2 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.327736 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23f5f085dd84d68beb277676225e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.327925 737 tablet_service.cc:1467] Processing CreateTablet for tablet 69012f97aac1441095aa3bb1548de9cd (DEFAULT_TABLE table=default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02 [id=543231f752ac46c39a13d6e8c4f46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.328258 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69012f97aac1441095aa3bb1548de9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.328225 754 tablet_service.cc:1467] Processing CreateTablet for tablet 443b29a93a0245e9a3a256e65a272699 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.328485 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443b29a93a0245e9a3a256e65a272699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.328661 745 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.328989 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9572179a6f45759214c6c39f22b2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.331110 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c058dcab55ca49298ff544dcb8c3f754 (DEFAULT_TABLE table=default.Table_99673c7bb11c4ff5a0b8879e673d3c3d [id=72335492497a49d4893aff0b17cf145b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.331434 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058dcab55ca49298ff544dcb8c3f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.332584 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7623bebb539e485aa508e95a35225979 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.332899 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7623bebb539e485aa508e95a35225979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.333137 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.333669 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.334067 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aeae6310362f4ec9bb90e693171c2319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.334484 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeae6310362f4ec9bb90e693171c2319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.334856 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9cf55e0a606d475ebb6261ecd57b9580"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.335266 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cf55e0a606d475ebb6261ecd57b9580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.336412 881 tablet_service.cc:1467] Processing CreateTablet for tablet 5311cd2a18654994a2fdb88be3e9490b (DEFAULT_TABLE table=default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb [id=ed24e86b77b444b095e2f85c3b1994b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.337167 875 tablet_service.cc:1467] Processing CreateTablet for tablet c23f5f085dd84d68beb277676225e7e2 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.337198 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd (DEFAULT_TABLE table=default.Table_28f569ded7ee456cb79ce58b8adb809d [id=c3a512398e9b4904a274b0f63bcde097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.337494 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.337744 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7623bebb539e485aa508e95a35225979 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.338119 878 tablet_service.cc:1467] Processing CreateTablet for tablet 69012f97aac1441095aa3bb1548de9cd (DEFAULT_TABLE table=default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02 [id=543231f752ac46c39a13d6e8c4f46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.338411 885 tablet_service.cc:1467] Processing CreateTablet for tablet 443b29a93a0245e9a3a256e65a272699 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.340950 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7c19fb4385ba42d8a7501cd54c098be9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.341485 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c19fb4385ba42d8a7501cd54c098be9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.344381 1164 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.344503 1164 ts_tablet_manager.cc:1397] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.143s user 0.002s sys 0.001s
I20250112 20:51:41.346606 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1dd19c189f4b1aa0faab3dac476a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.346972 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f52a9216c5044f88a0060e3e8e95eddd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.347267 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829e9dd5854961b5402b5710624612 with delete_type TABLET_DATA_DELETED (replaced by tablet 7623bebb539e485aa508e95a35225979) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.347122 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.347209 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.347479 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 40656387a5fd4c379ff2c6ef663df1a2 (DEFAULT_TABLE table=default.Table_ceb775b58bf04d71a6e9cec1b517610d [id=459ae6f5e4b14457b5bf8aa76114557f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.348062 1016 tablet_service.cc:1467] Processing CreateTablet for tablet d5afc68113174c179d33fac8f46e4de5 (DEFAULT_TABLE table=default.Table_632fa444ce9749b6a40c85532cb2e6c6 [id=df8bf02bb9b44292beebebe400c8667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.348214 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40656387a5fd4c379ff2c6ef663df1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.348438 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5afc68113174c179d33fac8f46e4de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.347340 1018 tablet_service.cc:1467] Processing CreateTablet for tablet c39a3858c52a4deabb019b6e2c3d4074 (DEFAULT_TABLE table=default.Table_8dd9ede7c9b34004b6ccc0e2e9043284 [id=5a7c2ba0e01543609d0ac3b612c0f30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.348870 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39a3858c52a4deabb019b6e2c3d4074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.353663 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet e2c41c2c92b442b986d6b75708c7420c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:41.353958 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 142119us
W20250112 20:51:41.355261 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 205820us
I20250112 20:51:41.355325 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:41.355612 710 leader_election.cc:343] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.355675 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9572179a6f45759214c6c39f22b2f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:41.356004 710 leader_election.cc:343] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:41.356351 710 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.356472 710 leader_election.cc:304] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:41.356719 2326 raft_consensus.cc:2743] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.347231 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b310e4135c2b424fb006a38a60e41066 with delete_type TABLET_DATA_DELETED (replaced by tablet aa9219fc971c4d15b0d383b20b61e540) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:41.357622 710 leader_election.cc:343] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:41.357920 710 leader_election.cc:343] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.347355 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 2527a0aaec5d441f8399cb2d2a6aa813 (DEFAULT_TABLE table=default.Table_f863419259044f8799ad66ba8d35c47e [id=87b4191b598f4360a6cd8ad6fe2de9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.360651 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2527a0aaec5d441f8399cb2d2a6aa813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.361492 1178 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.361613 1178 ts_tablet_manager.cc:1397] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.219s user 0.002s sys 0.000s
I20250112 20:51:41.361860 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93a5d375334552bc68510aebb507dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.362396 1178 raft_consensus.cc:357] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.362573 1178 raft_consensus.cc:383] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.362623 1178 raft_consensus.cc:738] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.362818 1178 consensus_queue.cc:260] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.363391 1178 ts_tablet_manager.cc:1428] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:41.367157 1185 ts_tablet_manager.cc:1918] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.367270 1185 log.cc:1198] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3bbb679585fc44699553fe85ea044566
I20250112 20:51:41.367630 1185 ts_tablet_manager.cc:1939] T 3bbb679585fc44699553fe85ea044566 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.369930 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f6995f7fbbf480995353b66749fbc10 with delete_type TABLET_DATA_DELETED (replaced by tablet 443b29a93a0245e9a3a256e65a272699) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.371001 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1a505986bc6146829cbafd0de6682d93
I20250112 20:51:41.371886 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cbdc5c117ca9437091a1667642c7c846
I20250112 20:51:41.372677 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 (DEFAULT_TABLE table=default.Table_c53abd31a1464c22abcb8d42a39cefac [id=af47182c306c4cc39ef2a02eba51d73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.373016 1029 tablet_service.cc:1467] Processing CreateTablet for tablet b040a8eefbe44fd49f6cf0bd2fdcf651 (DEFAULT_TABLE table=default.Table_7afee759e98341ce855fc733cb16a5f9 [id=b0880b74fadf49a2a202d0e7602b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.373134 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed925a5ef3a54f819ca292a4e7967b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.373328 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b040a8eefbe44fd49f6cf0bd2fdcf651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.373582 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 49bfc4fbe78745049699c8383c724f38
I20250112 20:51:41.373584 1010 tablet_service.cc:1467] Processing CreateTablet for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5 (DEFAULT_TABLE table=default.Table_e35d3834d4b64573afe5a5ab0cac708b [id=77be21069e934b91ab76bce3194366c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.373873 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5368ad6737e42c3bf9f9f3cf0be1fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.377004 1186 ts_tablet_manager.cc:1918] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.377115 1027 tablet_service.cc:1467] Processing CreateTablet for tablet beeaf42f79834d4f890aa6690c786562 (DEFAULT_TABLE table=default.Table_ecd375c288164374b6c8e4927fd9bc32 [id=c8bb5600a83d4d1890ddc10dd9f54d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.377463 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beeaf42f79834d4f890aa6690c786562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.377626 1186 log.cc:1198] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4ce4f31cf8f24e35b7281d4a8b5667a0
I20250112 20:51:41.377979 1186 ts_tablet_manager.cc:1939] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.381047 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 36f95046c87846b2aa7e15430c1480b5 (DEFAULT_TABLE table=default.Table_fc795f0197f74154a56a81c67c43185c [id=09f9919ce6db46dfb9a6527e7d7e7a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.381325 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f95046c87846b2aa7e15430c1480b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.381484 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 (DEFAULT_TABLE table=default.Table_a53e0c8faffe4744926220db2ddb69f9 [id=02b953cc10a3404391c5e29313552312]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.381844 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10f7a47089a43d8ae9c2fc16d379740. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.384467 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 45ac551767ed4e1c8da08e0e1177a7f2 (DEFAULT_TABLE table=default.Table_b9e4f2b60aef4766b60f2655988f3a57 [id=ef6dbfab1286406f81d97103dcb570e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.384748 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ac551767ed4e1c8da08e0e1177a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.391366 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d339bc4c8cd1481ebcfd721512976691 (DEFAULT_TABLE table=default.Table_45f57bdcc8a24f6695242226c7422b8c [id=2ca8265e8ae946f887c485f2bffff3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.391667 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d339bc4c8cd1481ebcfd721512976691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.396417 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b788acf8047acb34ca2596bbf141b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.397065 1026 tablet_service.cc:1467] Processing CreateTablet for tablet da93a5d375334552bc68510aebb507dd (DEFAULT_TABLE table=default.Table_2ac80cd3fa134256b29e1eb39796aa6d [id=e8c3c2cddc8042c692d8abe7a15aa05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.397398 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93a5d375334552bc68510aebb507dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.399355 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 (DEFAULT_TABLE table=default.Table_1de9e41db6aa48deb9875ff6f40a87bf [id=ec04d31413a04ffa9f1529189bb6a54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.399662 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961a72a9d8a643cea743f3f021bd5a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.400280 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a505986bc6146829cbafd0de6682d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.400461 891 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdc5c117ca9437091a1667642c7c846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.400498 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet f720bd9710d2476d9632699a2732cb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.401145 2466 tablet.cc:1721] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.403095 1163 tablet_bootstrap.cc:492] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:41.403223 1163 ts_tablet_manager.cc:1397] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.423s user 0.003s sys 0.000s
I20250112 20:51:41.403898 1163 raft_consensus.cc:357] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.404088 1163 raft_consensus.cc:383] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.404150 1163 raft_consensus.cc:738] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.404382 1163 consensus_queue.cc:260] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.406720 1163 ts_tablet_manager.cc:1428] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:41.407053 1163 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:41.407058 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bfc4fbe78745049699c8383c724f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.407416 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 (DEFAULT_TABLE table=default.Table_cdcbd0285dd04edebfa92654c16e5d39 [id=db95b74a655c4f57a166e73c8a6ed63a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.407747 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b062c1d67d4443eabe8ac1d8a832c243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.409129 1026 tablet_service.cc:1467] Processing CreateTablet for tablet fc1335522a384eb98163a2539630b93e (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.409451 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1335522a384eb98163a2539630b93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.412110 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc620e652af0401bb45d3cab54240f0c" } new_table_name: "default.Table_44a0d2b28ca7481881dcb3f0efdbb92a" modify_external_catalogs: true
I20250112 20:51:41.412624 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c83b256df0ed403ca72f7f240ce44c74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:41.413092 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5808d790defa44d7b6067522207c1022 (DEFAULT_TABLE table=default.Table_2ee684deb6634f31981eab910abbc0fb [id=891c37ab536643a5a84cce4f394c691c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:41.413208 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c83b256df0ed403ca72f7f240ce44c74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.413370 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5808d790defa44d7b6067522207c1022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.413604 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_aa8e6a73adec405db977cedb31479617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.414083 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa8e6a73adec405db977cedb31479617 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.418665 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 60468.9us
I20250112 20:51:41.418735 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_aeb009faaf764c289ecacebf52f2fcdd" } new_table_name: "default.Table_52386248ab844a8b81d154eadec805c7" modify_external_catalogs: true
W20250112 20:51:41.419018 710 leader_election.cc:343] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.419095 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_78c7d86c96c84c07b4baf2e7d8121a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.419281 710 leader_election.cc:343] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:41.419548 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78c7d86c96c84c07b4baf2e7d8121a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.419549 710 leader_election.cc:343] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.419628 710 leader_election.cc:304] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:41.419862 2326 raft_consensus.cc:2743] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.420084 2326 consensus_queue.cc:1035] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:41.420389 1164 raft_consensus.cc:357] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.420586 1164 raft_consensus.cc:383] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.420650 1164 raft_consensus.cc:738] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.420929 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a505986bc6146829cbafd0de6682d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:41.421052 710 consensus_peers.cc:487] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.421273 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bfc4fbe78745049699c8383c724f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.420933 1164 consensus_queue.cc:260] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.422484 1164 ts_tablet_manager.cc:1428] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.078s user 0.002s sys 0.000s
I20250112 20:51:41.422570 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ff8ac6f4d94c10b03a764bbf910c44" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:41.422747 772 raft_consensus.cc:3054] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.433127 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5311cd2a18654994a2fdb88be3e9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.433331 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1335522a384eb98163a2539630b93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.433857 772 raft_consensus.cc:2463] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:41.438045 1186 tablet_replica.cc:331] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.438222 1186 raft_consensus.cc:2238] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.438419 1186 pending_rounds.cc:62] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:41.438457 1186 pending_rounds.cc:69] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585051488350208 op_type: NO_OP noop_request { }
I20250112 20:51:41.438540 1186 raft_consensus.cc:2883] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.438607 1186 raft_consensus.cc:2267] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.439265 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23f5f085dd84d68beb277676225e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.439561 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443b29a93a0245e9a3a256e65a272699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.441093 1178 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.444242 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69012f97aac1441095aa3bb1548de9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.448935 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5808d790defa44d7b6067522207c1022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.449869 747 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdc5c117ca9437091a1667642c7c846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.452214 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7623bebb539e485aa508e95a35225979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.453629 2447 ts_tablet_manager.cc:1918] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.453735 2447 log.cc:1198] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4301a07fcb044fd2a6f3e5ef424b411f
I20250112 20:51:41.454080 2447 ts_tablet_manager.cc:1939] T 4301a07fcb044fd2a6f3e5ef424b411f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:41.454430 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5311cd2a18654994a2fdb88be3e9490b (DEFAULT_TABLE table=default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb [id=ed24e86b77b444b095e2f85c3b1994b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.454744 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5311cd2a18654994a2fdb88be3e9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.454912 2447 tablet_replica.cc:331] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.455076 2447 raft_consensus.cc:2238] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.455212 2447 raft_consensus.cc:2267] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.455650 2447 ts_tablet_manager.cc:1905] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.456614 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 7623bebb539e485aa508e95a35225979 (DEFAULT_TABLE table=default.Table_bc9eeca047a94f41a303c7de83574d30 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.421706 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3bbb679585fc44699553fe85ea044566 (table default.Table_8eeab516a6794571a8336068218e7a3a [id=2bc2a6f29dda4e13a781af77ba66b46f]) successfully deleted
W20250112 20:51:41.457022 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9c6d5d425e6a493daa46900cbd8db166 because the tablet was not found. No further retry: Not found: Tablet not found: 9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:41.458133 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c23f5f085dd84d68beb277676225e7e2 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.458585 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23f5f085dd84d68beb277676225e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.459134 1185 tablet_replica.cc:331] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.459353 1185 raft_consensus.cc:2238] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.459614 1185 raft_consensus.cc:2267] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.461869 1186 ts_tablet_manager.cc:1905] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.461825 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a6b14949b43a4bbd81fd8004f55380bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.462340 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6b14949b43a4bbd81fd8004f55380bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.462872 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" } new_table_name: "default.Table_f5d19e04b96b479d9c07282d7bb58644" modify_external_catalogs: true
I20250112 20:51:41.462886 2471 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:41.463059 2471 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:41.463385 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_248b046a301742db9d84c20c253dd4e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.463764 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_248b046a301742db9d84c20c253dd4e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.464128 1352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:41.464636 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (table default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]) successfully deleted
W20250112 20:51:41.464879 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3f0a808d431a43fb9299fec1dcf6d3b9 because the tablet was not found. No further retry: Not found: Tablet not found: 3f0a808d431a43fb9299fec1dcf6d3b9
I20250112 20:51:41.465704 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:41.466378 2031 leader_election.cc:304] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5c09e899184de6b770dca873501062; no voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:41.466645 1178 tablet_bootstrap.cc:654] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.460925 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 762388f687dd4b05ae6ffdaf8c64c999 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ff5efaf91174cfcbd56e2f23ae08ed0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.468353 1185 ts_tablet_manager.cc:1905] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.466195 1163 tablet_bootstrap.cc:654] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.472918 847 leader_election.cc:304] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:41.473323 2371 raft_consensus.cc:2798] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:41.473425 2371 raft_consensus.cc:695] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:41.473620 2371 consensus_queue.cc:237] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.481329 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9dde7822fdb743bda1e94189dd58b5ad" } modify_external_catalogs: true
I20250112 20:51:41.481547 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9dde7822fdb743bda1e94189dd58b5ad" } modify_external_catalogs: true
I20250112 20:51:41.481724 2443 raft_consensus.cc:491] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.481881 2443 raft_consensus.cc:513] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.482455 2443 leader_election.cc:290] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:41.483376 983 leader_election.cc:343] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4ce4f31cf8f24e35b7281d4a8b5667a0
I20250112 20:51:41.483685 2471 leader_election.cc:290] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271)
I20250112 20:51:41.483860 2471 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:41.486124 1178 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.486236 1178 ts_tablet_manager.cc:1397] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.003s
I20250112 20:51:41.486825 1178 raft_consensus.cc:357] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.487149 1178 raft_consensus.cc:383] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.487217 1178 raft_consensus.cc:738] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.487432 1178 consensus_queue.cc:260] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.487895 1178 ts_tablet_manager.cc:1428] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:41.488719 1178 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.489923 1178 tablet_bootstrap.cc:654] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:41.498605 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 140276b5b8c14512bb28b864bbbc3843 consensus state has no leader
W20250112 20:51:41.499334 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56544c3b0cf140089baa678765a2eb79 consensus state has no leader
I20250112 20:51:41.512813 1185 ts_tablet_manager.cc:1918] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.514318 1185 log.cc:1198] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:41.516037 1185 ts_tablet_manager.cc:1939] T d570daec845f40778c48ff3f2e2e30cb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:41.516741 847 consensus_peers.cc:487] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49a9e6cf325d48cc9a2a956400a68deb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.512758 2479 tablet.cc:1721] T 9cf1dc64f1964202bfa8169388a34781 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.517751 2466 tablet.cc:1721] T 9cf1dc64f1964202bfa8169388a34781 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.519227 2478 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.519810 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4f31cf8f24e35b7281d4a8b5667a0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:41.520038 770 raft_consensus.cc:2388] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:41.521981 1186 ts_tablet_manager.cc:1918] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.522145 1186 log.cc:1198] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:41.522500 1186 ts_tablet_manager.cc:1939] T 944b71897bcf4304bd9f7bb7adef34e5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.522930 1186 tablet_replica.cc:331] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.523149 1186 raft_consensus.cc:2238] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.524847 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7623bebb539e485aa508e95a35225979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.525938 1186 raft_consensus.cc:2267] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.526320 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet cbdc5c117ca9437091a1667642c7c846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.526479 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 (DEFAULT_TABLE table=default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.526896 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a505986bc6146829cbafd0de6682d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.526919 1186 ts_tablet_manager.cc:1905] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.521095 983 leader_election.cc:304] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.527256 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9572179a6f45759214c6c39f22b2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.529559 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 443b29a93a0245e9a3a256e65a272699 (DEFAULT_TABLE table=default.Table_9b0e136fb45d404eb7351439ce056024 [id=918ef48ed9e246f3ba9c7fd77fa7c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.530021 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443b29a93a0245e9a3a256e65a272699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.530181 1164 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.531638 1164 tablet_bootstrap.cc:654] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.532887 2479 tablet.cc:1721] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.535261 1185 tablet_replica.cc:331] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.535657 1185 raft_consensus.cc:2238] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.537142 1185 raft_consensus.cc:2267] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.537928 1185 ts_tablet_manager.cc:1905] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.539295 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 69012f97aac1441095aa3bb1548de9cd (DEFAULT_TABLE table=default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02 [id=543231f752ac46c39a13d6e8c4f46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.540778 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69012f97aac1441095aa3bb1548de9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.542749 2108 raft_consensus.cc:491] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.542882 2108 raft_consensus.cc:513] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.543483 2108 leader_election.cc:290] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:41.543871 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:41.548622 847 leader_election.cc:343] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.554756 1185 ts_tablet_manager.cc:1918] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.554877 1185 log.cc:1198] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:41.555192 1185 ts_tablet_manager.cc:1939] T f3ff8ac6f4d94c10b03a764bbf910c44 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.558743 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ab9be1f065ec4605a0a3fe69770793d0
I20250112 20:51:41.558976 1094 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:41.560539 1186 ts_tablet_manager.cc:1918] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.560721 1186 log.cc:1198] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:41.561128 1186 ts_tablet_manager.cc:1939] T 9c6d5d425e6a493daa46900cbd8db166 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.561136 2443 raft_consensus.cc:491] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.561300 2443 raft_consensus.cc:513] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.561883 2443 leader_election.cc:290] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.562014 2443 raft_consensus.cc:491] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.562077 1186 tablet_replica.cc:331] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.562144 2443 raft_consensus.cc:513] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.562306 1186 raft_consensus.cc:2238] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.562486 1186 raft_consensus.cc:2267] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.562646 2443 leader_election.cc:290] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.562904 2443 raft_consensus.cc:3054] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:41.565277 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7623bebb539e485aa508e95a35225979 consensus state has no leader
I20250112 20:51:41.565795 1186 ts_tablet_manager.cc:1905] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:41.568965 847 consensus_peers.cc:487] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49a9e6cf325d48cc9a2a956400a68deb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.569723 2443 raft_consensus.cc:2743] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.573640 1164 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.573861 1164 ts_tablet_manager.cc:1397] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250112 20:51:41.574590 1164 raft_consensus.cc:357] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.574894 1164 raft_consensus.cc:383] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.575002 1164 raft_consensus.cc:738] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.575307 1164 consensus_queue.cc:260] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.575986 1164 ts_tablet_manager.cc:1428] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:41.576742 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.578603 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bfc4fbe78745049699c8383c724f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.578832 1186 ts_tablet_manager.cc:1918] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.578940 1186 log.cc:1198] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/64887b3578284cb286ee122ab8c626c4
I20250112 20:51:41.579058 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.579308 1186 ts_tablet_manager.cc:1939] T 64887b3578284cb286ee122ab8c626c4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.579794 1186 tablet_replica.cc:331] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.579982 1186 raft_consensus.cc:2238] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.581673 878 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.581964 1186 raft_consensus.cc:2267] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.582794 1186 ts_tablet_manager.cc:1905] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.583011 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1a04c223889f424e831e82a52a84f804 (table default.Table_5b7e3278ff6a4aa8b3c9ab9b88366533 [id=76e117ae036e4e55b94608c9f10b76f7]) successfully deleted
I20250112 20:51:41.583688 1164 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.584077 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3b7f79b3309e45afb9e18cad97ddfeb0 (table default.Table_c5e95d792c48441db50a9e741756db91 [id=8c48e7960c62416c863975e795560a80]) successfully deleted
I20250112 20:51:41.585196 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.585709 2467 tablet.cc:1721] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:41.594259 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8c6a19ee7a4e4afab12870e48ee8df7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6a19ee7a4e4afab12870e48ee8df7f already in progress: creating tablet
I20250112 20:51:41.595757 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 49a9e6cf325d48cc9a2a956400a68deb (table default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]) successfully deleted
I20250112 20:51:41.596413 1185 tablet_replica.cc:331] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.596642 1185 raft_consensus.cc:2238] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.596870 1185 raft_consensus.cc:2267] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.597420 1185 ts_tablet_manager.cc:1905] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.607196 1164 tablet_bootstrap.cc:654] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.609874 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4301a07fcb044fd2a6f3e5ef424b411f (table default.Table_86c60a78bd264bc08c315269fdf3240f [id=b6a3dc0a355141b89067aea87042a20c]) successfully deleted
I20250112 20:51:41.611577 1185 ts_tablet_manager.cc:1918] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.611747 1185 log.cc:1198] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4ce4f31cf8f24e35b7281d4a8b5667a0
I20250112 20:51:41.612126 1185 ts_tablet_manager.cc:1939] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.615593 1186 ts_tablet_manager.cc:1918] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.615749 1186 log.cc:1198] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/49a9e6cf325d48cc9a2a956400a68deb
I20250112 20:51:41.616163 1186 ts_tablet_manager.cc:1939] T 49a9e6cf325d48cc9a2a956400a68deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.617303 1186 tablet_replica.cc:331] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.617560 1185 tablet_replica.cc:331] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.617606 1186 raft_consensus.cc:2238] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.617832 1185 raft_consensus.cc:2238] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.617872 1186 raft_consensus.cc:2267] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.618106 1185 pending_rounds.cc:62] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:41.618204 1185 pending_rounds.cc:69] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113585054585667584 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "d0829e9dd5854961b5402b5710624612" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" new_extra_config { } }
I20250112 20:51:41.618386 1186 ts_tablet_manager.cc:1905] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:41.618575 1185 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47160: Aborted: Op aborted
I20250112 20:51:41.622238 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d570daec845f40778c48ff3f2e2e30cb (table default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]) successfully deleted
W20250112 20:51:41.622617 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f52a9216c5044f88a0060e3e8e95eddd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52a9216c5044f88a0060e3e8e95eddd already in progress: creating tablet
I20250112 20:51:41.622913 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f3ff8ac6f4d94c10b03a764bbf910c44 (table default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]) successfully deleted
W20250112 20:51:41.623229 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4bd46a422093495ba8b5b03039874a32 because the tablet was not found. No further retry: Not found: Tablet not found: 4bd46a422093495ba8b5b03039874a32
I20250112 20:51:41.623492 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (table default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]) successfully deleted
I20250112 20:51:41.623724 1186 ts_tablet_manager.cc:1918] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:41.623751 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a23281d9b6f47a5bf627374fa0cfbf2 already in progress: creating tablet
W20250112 20:51:41.624046 440 catalog_manager.cc:4951] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): alter failed for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]): Aborted: Op aborted
I20250112 20:51:41.624820 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 725e56fa8ac343e5b1ec1214da02b7d4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:41.625777 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2767c4e6cdbe4f798cc72acefaa386f7" } new_table_name: "default.Table_7e40c8ac7b0842fb81f72ea44cda9bbd" modify_external_catalogs: true
I20250112 20:51:41.626356 1186 log.cc:1198] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:41.626768 1186 ts_tablet_manager.cc:1939] T 2469127891524be5ac3dc13fbcf35c7a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.627480 1186 tablet_replica.cc:331] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.627660 1185 raft_consensus.cc:2267] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.627878 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_06293bb20aca4d9696355a7fc3fdbe54" } modify_external_catalogs: true
I20250112 20:51:41.628082 1186 raft_consensus.cc:2238] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.628448 1186 raft_consensus.cc:2267] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.628890 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_06293bb20aca4d9696355a7fc3fdbe54" } modify_external_catalogs: true
I20250112 20:51:41.629349 1186 ts_tablet_manager.cc:1905] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.633126 1185 ts_tablet_manager.cc:1905] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.634330 2466 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.635454 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:41.635663 770 raft_consensus.cc:2463] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:41.635993 847 leader_election.cc:304] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:41.636036 710 consensus_peers.cc:487] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2469127891524be5ac3dc13fbcf35c7a. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.636247 2108 raft_consensus.cc:2798] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:41.636338 2108 raft_consensus.cc:491] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:41.636412 2108 raft_consensus.cc:3054] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:41.638589 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f6ad2ac50c745448031bdf94e0eea59 consensus state has no leader
I20250112 20:51:41.638788 2108 raft_consensus.cc:513] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.642843 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:41.643512 847 leader_election.cc:343] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.643699 1044 raft_consensus.cc:3054] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.643733 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f34aa32d6474561988b2cf3d0e8fd3b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:41.646064 2090 ts_manager.cc:194] Registered new tserver with Master: 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:41.646147 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 125057us
I20250112 20:51:41.647397 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:41.647562 2443 raft_consensus.cc:491] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.647711 2443 raft_consensus.cc:513] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.648346 2443 leader_election.cc:290] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.648931 1042 raft_consensus.cc:3054] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.649219 2108 leader_election.cc:290] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:41.649653 1163 tablet_bootstrap.cc:492] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:41.649807 1163 ts_tablet_manager.cc:1397] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.243s user 0.000s sys 0.008s
I20250112 20:51:41.650249 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:41.650656 1163 raft_consensus.cc:357] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.650857 1163 raft_consensus.cc:383] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.650928 1163 raft_consensus.cc:738] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.650888 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_60a304553d7548dc89dd2d4b1e5f4bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:41.651113 1163 consensus_queue.cc:260] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:41.651446 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60a304553d7548dc89dd2d4b1e5f4bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.651587 1163 ts_tablet_manager.cc:1428] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:41.651890 1163 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:41.649446 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:41.654019 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.654680 1178 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.654848 1178 ts_tablet_manager.cc:1397] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.167s user 0.028s sys 0.024s
I20250112 20:51:41.655174 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b519b0dc51a240629b6376767fc19e5d
I20250112 20:51:41.658247 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 53fcb48fe5314957ae807902274ef596) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.661156 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:41.661355 770 raft_consensus.cc:3054] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.664798 770 raft_consensus.cc:2463] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:41.665946 2108 raft_consensus.cc:491] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.666128 2108 raft_consensus.cc:513] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.666752 2108 leader_election.cc:290] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:41.667165 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570daec845f40778c48ff3f2e2e30cb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:41.667675 847 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.669636 2467 tablet.cc:1721] T 06253bb13a9541d8b408d1bf73e469dd P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.670969 2466 tablet.cc:1721] T 06253bb13a9541d8b408d1bf73e469dd P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.672680 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_91d16ee866374194947b9edddcf2e992" } new_table_name: "default.Table_ef26f7675c9640a7ac1673df48f024ac" modify_external_catalogs: true
I20250112 20:51:41.673578 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f720bd9710d2476d9632699a2732cb05" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:41.674235 1178 raft_consensus.cc:357] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.674413 1178 raft_consensus.cc:383] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.674469 1178 raft_consensus.cc:738] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.674644 1178 consensus_queue.cc:260] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.675143 1178 ts_tablet_manager.cc:1428] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250112 20:51:41.675387 1178 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.677457 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912d588104f482498c40239c9e585b9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:41.677953 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1c32cab2c4b438d3ff4a7e674feeb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:41.679651 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d44c14aead8249a291b194f228c5ca2b consensus state has no leader
I20250112 20:51:41.679940 1185 ts_tablet_manager.cc:1918] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:41.680110 1185 log.cc:1198] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d0829e9dd5854961b5402b5710624612
W20250112 20:51:41.680300 710 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.680466 710 leader_election.cc:304] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:41.680729 2326 raft_consensus.cc:2743] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.682507 847 leader_election.cc:304] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:41.682721 2108 raft_consensus.cc:2798] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:41.682821 2108 raft_consensus.cc:695] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:41.683044 2108 consensus_queue.cc:237] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.684283 983 leader_election.cc:304] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.684860 983 leader_election.cc:304] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.685497 983 leader_election.cc:304] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.686578 1164 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.686766 1164 ts_tablet_manager.cc:1397] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.002s
I20250112 20:51:41.687520 1164 raft_consensus.cc:357] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.687919 1164 raft_consensus.cc:383] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.688033 1164 raft_consensus.cc:738] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.680531 1185 ts_tablet_manager.cc:1939] T d0829e9dd5854961b5402b5710624612 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.688277 1164 consensus_queue.cc:260] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.688988 1164 ts_tablet_manager.cc:1428] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:41.685129 2488 raft_consensus.cc:2743] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:41.689553 1164 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.690929 1164 tablet_bootstrap.cc:654] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.691147 2485 raft_consensus.cc:2743] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.691602 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a64bcccf91b84c25b6a7cbae25eedc8a" } modify_external_catalogs: true
I20250112 20:51:41.691805 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a64bcccf91b84c25b6a7cbae25eedc8a" } modify_external_catalogs: true
I20250112 20:51:41.693234 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_19cd3e26ebaa41008374cb9290c3e323" } modify_external_catalogs: true
I20250112 20:51:41.693408 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_19cd3e26ebaa41008374cb9290c3e323" } modify_external_catalogs: true
I20250112 20:51:41.694149 1178 tablet_bootstrap.cc:654] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.694337 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a28d1068edaf4014a962ab336f9243d5" } modify_external_catalogs: true
I20250112 20:51:41.694506 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a28d1068edaf4014a962ab336f9243d5" } modify_external_catalogs: true
I20250112 20:51:41.695276 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c331f1ff143e43c99fac735c4547b362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.695825 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c331f1ff143e43c99fac735c4547b362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.697237 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1dff4cb9aa364c87b97f2d05462b70ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.697677 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dff4cb9aa364c87b97f2d05462b70ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:41.700466 847 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:41.700619 847 leader_election.cc:304] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.701431 2371 raft_consensus.cc:2743] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.701838 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_84c10c7cd6164c07aed6ccf21fe30476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.702412 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84c10c7cd6164c07aed6ccf21fe30476 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.702345 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d47332a374d74462aa73b72ac64269bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:41.702761 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d47332a374d74462aa73b72ac64269bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:41.711265 2447 ts_tablet_manager.cc:1918] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.711449 2447 log.cc:1198] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/81b932b7e1e34d4ab18aae40047dc038
I20250112 20:51:41.717046 2447 ts_tablet_manager.cc:1939] T 81b932b7e1e34d4ab18aae40047dc038 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:41.719031 2447 tablet_replica.cc:331] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.720223 2447 raft_consensus.cc:2238] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.720495 2447 raft_consensus.cc:2267] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.721212 2447 ts_tablet_manager.cc:1905] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.725128 1186 ts_tablet_manager.cc:1918] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.725296 1186 log.cc:1198] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:41.725565 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:41.725636 1186 ts_tablet_manager.cc:1939] T 35dcd1a120684446bbe19303fb8416ef P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.726716 1186 tablet_replica.cc:331] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.726980 1186 raft_consensus.cc:2238] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:41.727254 1186 pending_rounds.cc:62] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:41.727351 1186 pending_rounds.cc:69] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585055638081536 op_type: NO_OP noop_request { }
I20250112 20:51:41.727520 1186 raft_consensus.cc:2883] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.727651 1186 raft_consensus.cc:2267] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.739145 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_fbe3a476695f40ec8737a0e3e4ad731f [id=eba58b5d38e8443e842cca705dc51aeb] alter complete (version 1)
I20250112 20:51:41.744853 1178 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.691195 1185 tablet_replica.cc:331] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.745996 1185 raft_consensus.cc:2238] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:41.701315 710 consensus_peers.cc:487] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.707263 2208 raft_consensus.cc:491] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.746809 1178 ts_tablet_manager.cc:1397] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.007s
I20250112 20:51:41.746747 2208 raft_consensus.cc:513] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.747332 2208 leader_election.cc:290] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:41.748076 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 46796.2us
I20250112 20:51:41.750209 1185 pending_rounds.cc:62] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:41.750589 1185 pending_rounds.cc:69] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585052364599296 op_type: NO_OP noop_request { }
I20250112 20:51:41.751881 1185 raft_consensus.cc:2883] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:41.753106 1185 raft_consensus.cc:2267] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.754716 2479 tablet.cc:1721] T 06253bb13a9541d8b408d1bf73e469dd P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:41.757447 1185 ts_tablet_manager.cc:1905] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.766379 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 944b71897bcf4304bd9f7bb7adef34e5 (table default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]) successfully deleted
I20250112 20:51:41.773490 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:41.774334 710 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.774973 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9c6d5d425e6a493daa46900cbd8db166 (table default.Table_a9245c5997d7416fa75ebf6202856645 [id=942d944d07b146fbb1e30e593fa0a266]) successfully deleted
I20250112 20:51:41.776108 1178 raft_consensus.cc:357] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.776284 1178 raft_consensus.cc:383] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.776329 1178 raft_consensus.cc:738] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.776510 1178 consensus_queue.cc:260] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.777009 1178 ts_tablet_manager.cc:1428] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250112 20:51:41.777647 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 64887b3578284cb286ee122ab8c626c4 (table default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]) successfully deleted
I20250112 20:51:41.779562 1186 ts_tablet_manager.cc:1905] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.781812 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d3ea347fd2694016a3cd3721b0cab608
I20250112 20:51:41.783064 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b37e9e95c6b745ccaba5446b04abcc9d
I20250112 20:51:41.783896 455 catalog_manager.cc:5526] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.713510 1163 tablet_bootstrap.cc:654] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.784335 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]). Expected version 1 got 0
I20250112 20:51:41.755317 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:41.759102 2467 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:41.785346 710 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.785454 710 leader_election.cc:304] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:41.785667 2440 raft_consensus.cc:2743] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.791895 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4633e6870ccd4ce5b2458bb22155ba42
I20250112 20:51:41.793354 1178 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.796716 1178 tablet_bootstrap.cc:654] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.791676 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 49a9e6cf325d48cc9a2a956400a68deb (table default.Table_75837bdca38e4a358cb5c44071b13b26 [id=21766d589930422e9b400b1ba31ad416]) successfully deleted
I20250112 20:51:41.805928 2484 raft_consensus.cc:2743] T e912d588104f482498c40239c9e585b9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.806344 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_745bd9e4da3f419aa5aef5b16e9e9d58 [id=7e2c24519e17479fb708264c8543222b] alter complete (version 1)
W20250112 20:51:41.809453 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8c6a19ee7a4e4afab12870e48ee8df7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c6a19ee7a4e4afab12870e48ee8df7f already in progress: creating tablet
I20250112 20:51:41.811782 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_3c5be7fd026a4048b1e177fe5956ac0a [id=c7f63522332840b6bc6cd7462015ad3d] alter complete (version 1)
W20250112 20:51:41.813865 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8f6995f7fbbf480995353b66749fbc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f6995f7fbbf480995353b66749fbc10 already in progress: creating tablet
I20250112 20:51:41.814126 1178 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.810701 1163 tablet_bootstrap.cc:492] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:41.814299 1163 ts_tablet_manager.cc:1397] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.000s
I20250112 20:51:41.814332 1178 ts_tablet_manager.cc:1397] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250112 20:51:41.815083 1163 raft_consensus.cc:357] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.815426 1178 raft_consensus.cc:357] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.815562 1163 raft_consensus.cc:383] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.815641 1163 raft_consensus.cc:738] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.815691 1178 raft_consensus.cc:383] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.815837 1178 raft_consensus.cc:738] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.815878 1163 consensus_queue.cc:260] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:41.817202 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1f1dd19c189f4b1aa0faab3dac476a4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1dd19c189f4b1aa0faab3dac476a4a already in progress: creating tablet
W20250112 20:51:41.817512 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f52a9216c5044f88a0060e3e8e95eddd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52a9216c5044f88a0060e3e8e95eddd already in progress: creating tablet
I20250112 20:51:41.816188 1178 consensus_queue.cc:260] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.817751 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 2469127891524be5ac3dc13fbcf35c7a (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
I20250112 20:51:41.818095 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 35dcd1a120684446bbe19303fb8416ef (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
I20250112 20:51:41.818158 1163 ts_tablet_manager.cc:1428] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:41.818540 1163 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:41.819226 2326 raft_consensus.cc:491] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.819388 2326 raft_consensus.cc:513] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.820052 1178 ts_tablet_manager.cc:1428] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:41.820039 2326 leader_election.cc:290] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.820271 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_830127232d00465b966ceb3ecce56921 [id=0cfb42c9026845b3bcb98a4ee3ff14f7] alter complete (version 1)
I20250112 20:51:41.820714 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70653110aec74727acec9d63c2eb0f05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:41.820900 913 raft_consensus.cc:2463] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:41.821236 710 leader_election.cc:304] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:41.821327 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3eeb085e764941198ae7f653bc24757d because the tablet was not found. No further retry: Not found: Tablet not found: 3eeb085e764941198ae7f653bc24757d
I20250112 20:51:41.821657 2326 raft_consensus.cc:2798] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:41.821734 2326 raft_consensus.cc:491] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:41.821803 2326 raft_consensus.cc:3054] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.823318 1028 tablet_service.cc:1467] Processing CreateTablet for tablet d94c67881e954c0ebecf5c31de665ac0 (DEFAULT_TABLE table=default.Table_eb719443baaa46e49179af022eae1a4b [id=54ce1bfbecca4b36bc6fa53ec55ec830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.824291 2326 raft_consensus.cc:513] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.824563 753 tablet_service.cc:1467] Processing CreateTablet for tablet d94c67881e954c0ebecf5c31de665ac0 (DEFAULT_TABLE table=default.Table_eb719443baaa46e49179af022eae1a4b [id=54ce1bfbecca4b36bc6fa53ec55ec830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.824991 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c67881e954c0ebecf5c31de665ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.825073 1163 tablet_bootstrap.cc:654] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.825450 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70653110aec74727acec9d63c2eb0f05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:41.826169 710 leader_election.cc:343] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.826973 747 tablet_service.cc:1467] Processing CreateTablet for tablet d50c4d84934641cb8f649249729fe50d (DEFAULT_TABLE table=default.Table_40709325c312462982486fcd141ade99 [id=fdf62a292a254f7ead896ecc45109a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.827303 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50c4d84934641cb8f649249729fe50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.828210 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d50c4d84934641cb8f649249729fe50d (DEFAULT_TABLE table=default.Table_40709325c312462982486fcd141ade99 [id=fdf62a292a254f7ead896ecc45109a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.828330 2326 leader_election.cc:290] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:41.828451 1185 ts_tablet_manager.cc:1918] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.828531 1185 log.cc:1198] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:41.829033 1164 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.829068 878 tablet_service.cc:1467] Processing CreateTablet for tablet 7987e9912fd2475697b69f59074cdf8e (DEFAULT_TABLE table=default.Table_e4d7d365450448129def61e210e853ef [id=c8a92b89364743a69ad293075d630f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.829165 1164 ts_tablet_manager.cc:1397] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.140s user 0.002s sys 0.000s
I20250112 20:51:41.829277 892 tablet_service.cc:1467] Processing CreateTablet for tablet d94c67881e954c0ebecf5c31de665ac0 (DEFAULT_TABLE table=default.Table_eb719443baaa46e49179af022eae1a4b [id=54ce1bfbecca4b36bc6fa53ec55ec830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.829739 883 tablet_service.cc:1467] Processing CreateTablet for tablet d50c4d84934641cb8f649249729fe50d (DEFAULT_TABLE table=default.Table_40709325c312462982486fcd141ade99 [id=fdf62a292a254f7ead896ecc45109a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.829862 1164 raft_consensus.cc:357] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.830030 1164 raft_consensus.cc:383] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.829993 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 7987e9912fd2475697b69f59074cdf8e (DEFAULT_TABLE table=default.Table_e4d7d365450448129def61e210e853ef [id=c8a92b89364743a69ad293075d630f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.830086 1164 raft_consensus.cc:738] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.830431 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 (DEFAULT_TABLE table=default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b [id=fa9e065b2de74338bb052d33a7c099f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.830524 1164 consensus_queue.cc:260] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.829051 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7987e9912fd2475697b69f59074cdf8e (DEFAULT_TABLE table=default.Table_e4d7d365450448129def61e210e853ef [id=c8a92b89364743a69ad293075d630f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.830720 885 tablet_service.cc:1467] Processing CreateTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 (DEFAULT_TABLE table=default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b [id=fa9e065b2de74338bb052d33a7c099f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.830971 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987e9912fd2475697b69f59074cdf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.831048 1164 ts_tablet_manager.cc:1428] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:41.833695 875 tablet_service.cc:1467] Processing CreateTablet for tablet 78ebe38565ad4192bdd6ebdb207b8479 (DEFAULT_TABLE table=default.Table_5df2936b4476476fa09325d619af0a27 [id=a2b8e6f3989041dc8d2293c8e47a5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.833716 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70653110aec74727acec9d63c2eb0f05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:41.834050 913 raft_consensus.cc:3054] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:41.836587 913 raft_consensus.cc:2463] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:41.836874 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 78ebe38565ad4192bdd6ebdb207b8479 (DEFAULT_TABLE table=default.Table_5df2936b4476476fa09325d619af0a27 [id=a2b8e6f3989041dc8d2293c8e47a5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.835296 747 tablet_service.cc:1467] Processing CreateTablet for tablet 78ebe38565ad4192bdd6ebdb207b8479 (DEFAULT_TABLE table=default.Table_5df2936b4476476fa09325d619af0a27 [id=a2b8e6f3989041dc8d2293c8e47a5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.837997 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ebe38565ad4192bdd6ebdb207b8479. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.838546 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987e9912fd2475697b69f59074cdf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.838613 881 tablet_service.cc:1467] Processing CreateTablet for tablet fa4e7313e9934f50816aba955976311f (DEFAULT_TABLE table=default.Table_5a2776514bca4b24ade2fab5a7ed71df [id=c017d9bff96346bba6b6728001c6d205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.838943 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa4e7313e9934f50816aba955976311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.841053 1018 tablet_service.cc:1467] Processing CreateTablet for tablet fa4e7313e9934f50816aba955976311f (DEFAULT_TABLE table=default.Table_5a2776514bca4b24ade2fab5a7ed71df [id=c017d9bff96346bba6b6728001c6d205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.842089 2108 raft_consensus.cc:491] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:41.842196 2108 raft_consensus.cc:513] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.842768 2108 leader_election.cc:290] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:41.843124 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ebb3c30194d9fa6a092d9bec4eadf" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:41.830664 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 (DEFAULT_TABLE table=default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b [id=fa9e065b2de74338bb052d33a7c099f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.837543 710 leader_election.cc:304] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:41.843590 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b299e16586145ee9ed5a7b118c049b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.844218 2326 raft_consensus.cc:2798] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:41.844651 2326 raft_consensus.cc:695] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:41.844949 2326 consensus_queue.cc:237] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.845104 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c67881e954c0ebecf5c31de665ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.846016 1164 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.847512 1164 tablet_bootstrap.cc:654] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.848996 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50c4d84934641cb8f649249729fe50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.848997 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94c67881e954c0ebecf5c31de665ac0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:41.849359 847 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:41.849438 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b299e16586145ee9ed5a7b118c049b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:41.849534 847 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.849613 847 leader_election.cc:304] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:41.849843 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa4e7313e9934f50816aba955976311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.850178 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ebe38565ad4192bdd6ebdb207b8479. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.850504 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50c4d84934641cb8f649249729fe50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.850895 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7987e9912fd2475697b69f59074cdf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.853013 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b299e16586145ee9ed5a7b118c049b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.853202 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ebe38565ad4192bdd6ebdb207b8479. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.861323 2108 raft_consensus.cc:2743] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:41.864120 1164 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.864243 1164 ts_tablet_manager.cc:1397] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20250112 20:51:41.866509 875 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.867125 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.869753 1164 raft_consensus.cc:357] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.870994 1164 raft_consensus.cc:383] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.871116 1164 raft_consensus.cc:738] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.871353 1164 consensus_queue.cc:260] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.873364 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.875545 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.876621 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b519b0dc51a240629b6376767fc19e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.877991 1164 ts_tablet_manager.cc:1428] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:41.878274 1185 ts_tablet_manager.cc:1939] T 2469127891524be5ac3dc13fbcf35c7a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.878228 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b519b0dc51a240629b6376767fc19e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.879870 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 64debf76654d47c4bfabe2e3717212a0 (DEFAULT_TABLE table=default.Table_78c7d86c96c84c07b4baf2e7d8121a20 [id=9aca2d3728a141b3b73ddbf074209b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.880167 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64debf76654d47c4bfabe2e3717212a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.881768 747 tablet_service.cc:1467] Processing CreateTablet for tablet 64debf76654d47c4bfabe2e3717212a0 (DEFAULT_TABLE table=default.Table_78c7d86c96c84c07b4baf2e7d8121a20 [id=9aca2d3728a141b3b73ddbf074209b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.882102 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64debf76654d47c4bfabe2e3717212a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.882869 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9208eb3d4a734e02a04f0ce19cecb94f (DEFAULT_TABLE table=default.Table_248b046a301742db9d84c20c253dd4e3 [id=49ae6de102ef45dc98b597c084dbcc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.883188 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208eb3d4a734e02a04f0ce19cecb94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.886483 1164 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.887984 1164 tablet_bootstrap.cc:654] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.888898 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 9208eb3d4a734e02a04f0ce19cecb94f (DEFAULT_TABLE table=default.Table_248b046a301742db9d84c20c253dd4e3 [id=49ae6de102ef45dc98b597c084dbcc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.889271 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208eb3d4a734e02a04f0ce19cecb94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.889376 1185 tablet_replica.cc:331] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:41.889899 741 tablet_service.cc:1467] Processing CreateTablet for tablet 25afa9455b3344e387901594252002ba (DEFAULT_TABLE table=default.Table_c83b256df0ed403ca72f7f240ce44c74 [id=bd9454b08964474fbe03cb3fc931192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.893316 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25afa9455b3344e387901594252002ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.893574 878 tablet_service.cc:1467] Processing CreateTablet for tablet 64debf76654d47c4bfabe2e3717212a0 (DEFAULT_TABLE table=default.Table_78c7d86c96c84c07b4baf2e7d8121a20 [id=9aca2d3728a141b3b73ddbf074209b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.894116 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64debf76654d47c4bfabe2e3717212a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.894411 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0 (DEFAULT_TABLE table=default.Table_be64887db24d47379842a579e8c4d332 [id=f67f26e3c7994ddc90ffb1372f9aede8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.896049 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 25afa9455b3344e387901594252002ba (DEFAULT_TABLE table=default.Table_c83b256df0ed403ca72f7f240ce44c74 [id=bd9454b08964474fbe03cb3fc931192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.893816 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0d072ebae78e40c48e158eb65775476c (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.896418 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25afa9455b3344e387901594252002ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.896806 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d072ebae78e40c48e158eb65775476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.897331 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 0d072ebae78e40c48e158eb65775476c (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.897802 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d072ebae78e40c48e158eb65775476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.899206 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff5efaf91174cfcbd56e2f23ae08ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.900470 1185 raft_consensus.cc:2238] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.900918 1185 raft_consensus.cc:2267] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.901917 1185 ts_tablet_manager.cc:1905] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.902166 744 tablet_service.cc:1467] Processing CreateTablet for tablet fa4e7313e9934f50816aba955976311f (DEFAULT_TABLE table=default.Table_5a2776514bca4b24ade2fab5a7ed71df [id=c017d9bff96346bba6b6728001c6d205]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.902539 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa4e7313e9934f50816aba955976311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.903124 1164 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.903311 1164 ts_tablet_manager.cc:1397] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20250112 20:51:41.904155 1164 raft_consensus.cc:357] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.904564 1164 raft_consensus.cc:383] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.904665 1164 raft_consensus.cc:738] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.904897 1164 consensus_queue.cc:260] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:41.907558 741 tablet_service.cc:1467] Processing CreateTablet for tablet 869e6594c48b4780b8e67d512159768e (DEFAULT_TABLE table=default.Table_9cf55e0a606d475ebb6261ecd57b9580 [id=2cb9219bd64042f9abfe6ee81f5b87f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.908142 881 tablet_service.cc:1467] Processing CreateTablet for tablet 25afa9455b3344e387901594252002ba (DEFAULT_TABLE table=default.Table_c83b256df0ed403ca72f7f240ce44c74 [id=bd9454b08964474fbe03cb3fc931192b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.908540 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25afa9455b3344e387901594252002ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.909844 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 869e6594c48b4780b8e67d512159768e (DEFAULT_TABLE table=default.Table_9cf55e0a606d475ebb6261ecd57b9580 [id=2cb9219bd64042f9abfe6ee81f5b87f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.910247 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70653110aec74727acec9d63c2eb0f05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:41.910813 710 leader_election.cc:343] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:41.911506 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869e6594c48b4780b8e67d512159768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.913484 1164 ts_tablet_manager.cc:1428] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:41.918520 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869e6594c48b4780b8e67d512159768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.919469 748 tablet_service.cc:1467] Processing CreateTablet for tablet d153d6ce727a4819ba9c409f9b87552e (DEFAULT_TABLE table=default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29 [id=e084d844ac1048799cd7333e1f5530cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.919898 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d153d6ce727a4819ba9c409f9b87552e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.920181 756 tablet_service.cc:1467] Processing CreateTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.920161 883 tablet_service.cc:1467] Processing CreateTablet for tablet d153d6ce727a4819ba9c409f9b87552e (DEFAULT_TABLE table=default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29 [id=e084d844ac1048799cd7333e1f5530cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.920469 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d153d6ce727a4819ba9c409f9b87552e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.920962 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 91a52a8471444db88f2d749000dae4e8 (DEFAULT_TABLE table=default.Table_aeae6310362f4ec9bb90e693171c2319 [id=774c46ee90bb4f4482608ff18c1da78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.921311 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a52a8471444db88f2d749000dae4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.922837 1178 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:41.924482 1178 tablet_bootstrap.cc:654] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.925685 882 tablet_service.cc:1467] Processing CreateTablet for tablet 91a52a8471444db88f2d749000dae4e8 (DEFAULT_TABLE table=default.Table_aeae6310362f4ec9bb90e693171c2319 [id=774c46ee90bb4f4482608ff18c1da78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.926019 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a52a8471444db88f2d749000dae4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.926195 745 tablet_service.cc:1467] Processing CreateTablet for tablet 42c88852d29545c7a937ca389929d2b1 (DEFAULT_TABLE table=default.Table_aa8e6a73adec405db977cedb31479617 [id=57fff778a5074760b1c2f42d1ef660b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.926445 884 tablet_service.cc:1467] Processing CreateTablet for tablet 42c88852d29545c7a937ca389929d2b1 (DEFAULT_TABLE table=default.Table_aa8e6a73adec405db977cedb31479617 [id=57fff778a5074760b1c2f42d1ef660b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.926528 737 tablet_service.cc:1467] Processing CreateTablet for tablet 91a52a8471444db88f2d749000dae4e8 (DEFAULT_TABLE table=default.Table_aeae6310362f4ec9bb90e693171c2319 [id=774c46ee90bb4f4482608ff18c1da78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.926757 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c88852d29545c7a937ca389929d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.926864 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a0679bf1ed4bbc85f14f62dfcc114d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.928002 2447 ts_tablet_manager.cc:1918] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.928100 2447 log.cc:1198] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b310e4135c2b424fb006a38a60e41066
I20250112 20:51:41.928380 2447 ts_tablet_manager.cc:1939] T b310e4135c2b424fb006a38a60e41066 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:41.929039 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 42c88852d29545c7a937ca389929d2b1 (DEFAULT_TABLE table=default.Table_aa8e6a73adec405db977cedb31479617 [id=57fff778a5074760b1c2f42d1ef660b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.929356 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c88852d29545c7a937ca389929d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.931749 885 tablet_service.cc:1467] Processing CreateTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.932056 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a0679bf1ed4bbc85f14f62dfcc114d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.933086 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a52a8471444db88f2d749000dae4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.933348 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b519b0dc51a240629b6376767fc19e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:41.936179 710 consensus_peers.cc:487] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:41.936980 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c88852d29545c7a937ca389929d2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.939926 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea347fd2694016a3cd3721b0cab608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.940316 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea347fd2694016a3cd3721b0cab608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.941529 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.942157 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.896225 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0d072ebae78e40c48e158eb65775476c (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.944265 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.897428 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9208eb3d4a734e02a04f0ce19cecb94f (DEFAULT_TABLE table=default.Table_248b046a301742db9d84c20c253dd4e3 [id=49ae6de102ef45dc98b597c084dbcc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.944685 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet b37e9e95c6b745ccaba5446b04abcc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:41.944866 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208eb3d4a734e02a04f0ce19cecb94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.944906 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.946081 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.947309 1186 ts_tablet_manager.cc:1918] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:41.947454 1186 log.cc:1198] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:41.947890 1186 ts_tablet_manager.cc:1939] T f3ff8ac6f4d94c10b03a764bbf910c44 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:41.948863 1186 tablet_replica.cc:331] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:41.949555 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 4633e6870ccd4ce5b2458bb22155ba42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:41.950083 1164 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.950850 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d072ebae78e40c48e158eb65775476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.952170 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 81b932b7e1e34d4ab18aae40047dc038 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
W20250112 20:51:41.952420 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1f1dd19c189f4b1aa0faab3dac476a4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1dd19c189f4b1aa0faab3dac476a4a already in progress: creating tablet
W20250112 20:51:41.952643 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f52a9216c5044f88a0060e3e8e95eddd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52a9216c5044f88a0060e3e8e95eddd already in progress: creating tablet
W20250112 20:51:41.952854 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d0829e9dd5854961b5402b5710624612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0829e9dd5854961b5402b5710624612 already in progress: creating tablet
W20250112 20:51:41.953064 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a23281d9b6f47a5bf627374fa0cfbf2 already in progress: creating tablet
W20250112 20:51:41.953246 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 43d341f6ac6848d9a8e60f689dd24a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43d341f6ac6848d9a8e60f689dd24a66 already in progress: creating tablet
W20250112 20:51:41.953452 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e2c41c2c92b442b986d6b75708c7420c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c41c2c92b442b986d6b75708c7420c already in progress: creating tablet
I20250112 20:51:41.954249 1164 tablet_bootstrap.cc:654] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.954816 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.956550 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a0679bf1ed4bbc85f14f62dfcc114d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.958503 1163 tablet_bootstrap.cc:492] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:41.958680 1163 ts_tablet_manager.cc:1397] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.000s
I20250112 20:51:41.956205 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d153d6ce727a4819ba9c409f9b87552e (DEFAULT_TABLE table=default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29 [id=e084d844ac1048799cd7333e1f5530cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.961230 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d153d6ce727a4819ba9c409f9b87552e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.967571 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d0829e9dd5854961b5402b5710624612 (table default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
I20250112 20:51:41.968317 1164 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:41.968318 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 2469127891524be5ac3dc13fbcf35c7a (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
I20250112 20:51:41.968748 1164 ts_tablet_manager.cc:1397] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250112 20:51:41.969195 2447 tablet_replica.cc:331] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:41.969372 2447 raft_consensus.cc:2238] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.969519 2447 raft_consensus.cc:2267] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:41.969815 1164 raft_consensus.cc:357] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.970091 1164 raft_consensus.cc:383] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.970192 1164 raft_consensus.cc:738] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.969369 1163 raft_consensus.cc:357] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.970425 1164 consensus_queue.cc:260] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.970640 1163 raft_consensus.cc:383] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.970718 1163 raft_consensus.cc:738] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.970906 1164 ts_tablet_manager.cc:1428] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:41.970925 1163 consensus_queue.cc:260] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:41.971590 1185 ts_tablet_manager.cc:1918] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:41.971679 1185 log.cc:1198] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:41.971961 1185 ts_tablet_manager.cc:1939] T 4f34aa32d6474561988b2cf3d0e8fd3b P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:41.973136 1164 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:41.975628 1164 tablet_bootstrap.cc:654] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:41.981916 2326 consensus_queue.cc:1035] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:41.983367 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3f6ad2ac50c745448031bdf94e0eea59 (table default.Table_7e40c8ac7b0842fb81f72ea44cda9bbd [id=b56651754d344c37913ce7942cd314fe], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3f6ad2ac50c745448031bdf94e0eea59 consensus state has no leader
I20250112 20:51:41.954200 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ea347fd2694016a3cd3721b0cab608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:41.954488 892 tablet_service.cc:1467] Processing CreateTablet for tablet 869e6594c48b4780b8e67d512159768e (DEFAULT_TABLE table=default.Table_9cf55e0a606d475ebb6261ecd57b9580 [id=2cb9219bd64042f9abfe6ee81f5b87f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:41.971545 1178 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:41.985093 1178 ts_tablet_manager.cc:1397] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.003s
I20250112 20:51:41.985235 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869e6594c48b4780b8e67d512159768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:41.985867 1186 raft_consensus.cc:2238] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:41.985899 1178 raft_consensus.cc:357] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.986075 1178 raft_consensus.cc:383] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:41.986147 1178 raft_consensus.cc:738] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:41.986420 2447 ts_tablet_manager.cc:1905] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:41.986369 1178 consensus_queue.cc:260] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:41.990993 1163 ts_tablet_manager.cc:1428] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20250112 20:51:42.001349 1178 ts_tablet_manager.cc:1428] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250112 20:51:42.001673 1178 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.002676 1186 raft_consensus.cc:2267] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.005518 1178 tablet_bootstrap.cc:654] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.009007 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 36935e6d8c5643388ad163458e2b387c (table default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51]). Expected version 1 got 0
I20250112 20:51:42.009263 1186 ts_tablet_manager.cc:1905] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.014516 1163 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.017194 2447 ts_tablet_manager.cc:1918] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.017304 2447 log.cc:1198] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f720bd9710d2476d9632699a2732cb05
I20250112 20:51:42.021471 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 64e1c32cab2c4b438d3ff4a7e674feeb from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:42.027079 2059 raft_consensus.cc:491] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.027262 2059 raft_consensus.cc:513] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.027936 2059 leader_election.cc:290] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.029158 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:42.029404 913 raft_consensus.cc:2463] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:42.030026 710 leader_election.cc:304] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:42.030606 2059 raft_consensus.cc:2798] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.030699 2059 raft_consensus.cc:491] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.030776 2059 raft_consensus.cc:3054] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.031865 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.033111 2447 ts_tablet_manager.cc:1939] T f720bd9710d2476d9632699a2732cb05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:42.033459 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:42.034068 455 catalog_manager.cc:6596] ReplaceTablet: tablet 64e1c32cab2c4b438d3ff4a7e674feeb deleted and replaced by tablet 23dc8a695ccd4f88bd108304add01a4e
W20250112 20:51:42.034368 710 leader_election.cc:343] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.035758 2447 tablet_replica.cc:331] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:42.036011 2447 raft_consensus.cc:2238] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.036202 2447 raft_consensus.cc:2267] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.036437 2059 raft_consensus.cc:513] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.036734 2447 ts_tablet_manager.cc:1905] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.037320 2059 leader_election.cc:290] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.037933 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.038389 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:42.038631 913 raft_consensus.cc:3054] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.039793 1163 tablet_bootstrap.cc:654] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.042444 913 raft_consensus.cc:2463] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:42.042964 710 leader_election.cc:304] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:42.043498 710 leader_election.cc:343] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.044004 2059 raft_consensus.cc:2798] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.046813 2059 raft_consensus.cc:695] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:42.047204 2059 consensus_queue.cc:237] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.053640 1163 tablet_bootstrap.cc:492] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.053779 1163 ts_tablet_manager.cc:1397] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250112 20:51:42.054466 1163 raft_consensus.cc:357] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.054647 1163 raft_consensus.cc:383] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.054692 1163 raft_consensus.cc:738] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.055436 1178 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.055579 1178 ts_tablet_manager.cc:1397] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.005s
I20250112 20:51:42.056252 1178 raft_consensus.cc:357] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.056417 1178 raft_consensus.cc:383] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.056457 1178 raft_consensus.cc:738] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.056671 1178 consensus_queue.cc:260] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.057209 1178 ts_tablet_manager.cc:1428] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:42.057709 1178 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.059540 2447 ts_tablet_manager.cc:1918] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:42.059636 2447 log.cc:1198] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4ce4f31cf8f24e35b7281d4a8b5667a0
I20250112 20:51:42.059688 1178 tablet_bootstrap.cc:654] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.059937 2447 ts_tablet_manager.cc:1939] T 4ce4f31cf8f24e35b7281d4a8b5667a0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:42.067127 1186 ts_tablet_manager.cc:1918] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:42.067351 1163 consensus_queue.cc:260] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.068025 1163 ts_tablet_manager.cc:1428] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 20:51:42.073056 1163 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.075920 1163 tablet_bootstrap.cc:654] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.076263 1186 log.cc:1198] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3bbb679585fc44699553fe85ea044566
I20250112 20:51:42.082847 1186 ts_tablet_manager.cc:1939] T 3bbb679585fc44699553fe85ea044566 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:42.083645 1186 tablet_replica.cc:331] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:42.085753 1163 tablet_bootstrap.cc:492] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.085944 1163 ts_tablet_manager.cc:1397] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250112 20:51:42.086580 1163 raft_consensus.cc:357] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.083810 1178 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.091475 2371 raft_consensus.cc:491] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.091514 1178 ts_tablet_manager.cc:1397] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20250112 20:51:42.091580 1186 raft_consensus.cc:2238] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.091730 1186 pending_rounds.cc:62] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:42.091612 2371 raft_consensus.cc:513] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.091773 1186 pending_rounds.cc:69] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113585054585667584 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "d0829e9dd5854961b5402b5710624612" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_bc9eeca047a94f41a303c7de83574d30" dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" new_extra_config { } }
I20250112 20:51:42.092060 1186 raft_consensus.cc:2267] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.092208 2371 leader_election.cc:290] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.092227 1178 raft_consensus.cc:357] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.092569 1163 raft_consensus.cc:383] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.092638 1163 raft_consensus.cc:738] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.092633 1178 raft_consensus.cc:383] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.092700 1178 raft_consensus.cc:738] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.092978 1178 consensus_queue.cc:260] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.093586 1178 ts_tablet_manager.cc:1428] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:42.093924 1178 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.092847 1163 consensus_queue.cc:260] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.094624 1163 ts_tablet_manager.cc:1428] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250112 20:51:42.095782 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05d287d81c46558acdd9a42edda6f8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.096323 1186 ts_tablet_manager.cc:1905] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.096925 1163 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.098230 1163 tablet_bootstrap.cc:654] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.099057 2108 raft_consensus.cc:491] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:42.099116 847 leader_election.cc:343] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.099177 2108 raft_consensus.cc:513] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.099699 2108 leader_election.cc:290] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.101217 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a4b5ac2f94874b99b3ef790a034fd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.100759 1178 tablet_bootstrap.cc:654] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.101416 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05d287d81c46558acdd9a42edda6f8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.101617 770 raft_consensus.cc:2463] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.101922 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a4b5ac2f94874b99b3ef790a034fd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.102097 774 raft_consensus.cc:2463] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:42.102209 847 leader_election.cc:343] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.104378 1163 tablet_bootstrap.cc:492] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.104583 1163 ts_tablet_manager.cc:1397] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250112 20:51:42.105405 1163 raft_consensus.cc:357] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.105643 1163 raft_consensus.cc:383] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.105746 1163 raft_consensus.cc:738] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.106251 847 leader_election.cc:304] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.106510 2108 raft_consensus.cc:2798] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.106588 2108 raft_consensus.cc:491] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.106009 1163 consensus_queue.cc:260] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.106649 2108 raft_consensus.cc:3054] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.106664 847 leader_election.cc:304] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.107013 2371 raft_consensus.cc:2798] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.107092 2371 raft_consensus.cc:491] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.107156 2371 raft_consensus.cc:3054] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.107252 1163 ts_tablet_manager.cc:1428] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:42.107486 1163 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.108896 1163 tablet_bootstrap.cc:654] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.109117 2108 raft_consensus.cc:513] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.109611 2108 leader_election.cc:290] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.110069 2371 raft_consensus.cc:513] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.110872 2371 leader_election.cc:290] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.111611 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05d287d81c46558acdd9a42edda6f8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.111764 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc05d287d81c46558acdd9a42edda6f8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.111925 774 raft_consensus.cc:3054] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.111932 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a4b5ac2f94874b99b3ef790a034fd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:42.112298 847 leader_election.cc:343] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:42.112460 847 leader_election.cc:343] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.112582 2371 consensus_queue.cc:1035] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:42.114418 774 raft_consensus.cc:2463] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.114683 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1a4b5ac2f94874b99b3ef790a034fd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.114846 770 raft_consensus.cc:3054] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.114776 847 leader_election.cc:304] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.116750 2108 raft_consensus.cc:2798] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.116873 2108 raft_consensus.cc:695] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.117127 2108 consensus_queue.cc:237] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.117343 770 raft_consensus.cc:2463] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.118178 1163 tablet_bootstrap.cc:492] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.118306 1163 ts_tablet_manager.cc:1397] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250112 20:51:42.119072 1163 raft_consensus.cc:357] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.119256 1163 raft_consensus.cc:383] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.119246 847 leader_election.cc:304] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.119304 1163 raft_consensus.cc:738] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.119498 1163 consensus_queue.cc:260] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.119637 2371 raft_consensus.cc:2798] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.119876 2371 raft_consensus.cc:695] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.120218 2371 consensus_queue.cc:237] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.121902 1163 ts_tablet_manager.cc:1428] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:42.122169 1163 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.123580 1163 tablet_bootstrap.cc:654] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.129346 1163 tablet_bootstrap.cc:492] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.129470 1163 ts_tablet_manager.cc:1397] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:42.130144 1163 raft_consensus.cc:357] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.130321 1163 raft_consensus.cc:383] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.130375 1163 raft_consensus.cc:738] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.130599 1163 consensus_queue.cc:260] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.133162 1163 ts_tablet_manager.cc:1428] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:42.133443 1163 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.134889 1163 tablet_bootstrap.cc:654] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.135270 1186 ts_tablet_manager.cc:1918] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:42.137980 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8fde83dfa40f4a8783c855991337d74a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.138870 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fde83dfa40f4a8783c855991337d74a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.141178 1163 tablet_bootstrap.cc:492] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.141294 1163 ts_tablet_manager.cc:1397] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250112 20:51:42.141947 1163 raft_consensus.cc:357] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.142123 1163 raft_consensus.cc:383] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.142174 1163 raft_consensus.cc:738] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.142361 1163 consensus_queue.cc:260] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.144857 1163 ts_tablet_manager.cc:1428] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:42.145174 1163 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.145802 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f53bd3309a434176b88eb5b8b727b344" } new_table_name: "default.Table_a6c9111b483148718ccd2bfb2a54698b" modify_external_catalogs: true
I20250112 20:51:42.151402 1163 tablet_bootstrap.cc:654] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:42.152024 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89f9b46da06a476ab9f2a93ca74ff736 consensus state has no leader
I20250112 20:51:42.135994 1178 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.156416 1178 ts_tablet_manager.cc:1397] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20250112 20:51:42.156292 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a38079a1fd8c479a839b50c4c84eca80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.157003 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a38079a1fd8c479a839b50c4c84eca80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.157207 1178 raft_consensus.cc:357] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.157367 1178 raft_consensus.cc:383] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.157409 1178 raft_consensus.cc:738] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.157569 1178 consensus_queue.cc:260] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.157931 1178 ts_tablet_manager.cc:1428] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:42.158179 1178 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.159775 1178 tablet_bootstrap.cc:654] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.160496 2415 raft_consensus.cc:491] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.160609 2415 raft_consensus.cc:513] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.161188 2415 leader_election.cc:290] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.162770 1186 log.cc:1198] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d0829e9dd5854961b5402b5710624612
I20250112 20:51:42.163218 1186 ts_tablet_manager.cc:1939] T d0829e9dd5854961b5402b5710624612 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:42.163865 1186 tablet_replica.cc:331] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:42.165575 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.166008 847 leader_election.cc:343] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.166395 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.166589 770 raft_consensus.cc:2463] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.166956 847 leader_election.cc:304] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
W20250112 20:51:42.171447 710 consensus_peers.cc:487] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:42.172035 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0b49521c49204a03b257a7b38a4132c2" } new_table_name: "default.Table_185f9bae27844411aa65dad64f12594e" modify_external_catalogs: true
I20250112 20:51:42.172854 2108 raft_consensus.cc:2798] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.172986 2108 raft_consensus.cc:491] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.173110 2108 raft_consensus.cc:3054] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.174929 2484 raft_consensus.cc:491] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.175079 2484 raft_consensus.cc:513] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.175679 2484 leader_election.cc:290] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.175747 2108 raft_consensus.cc:513] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.176342 2108 leader_election.cc:290] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.176465 2108 raft_consensus.cc:491] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.176560 2108 raft_consensus.cc:513] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.177040 2108 leader_election.cc:290] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.177475 1163 tablet_bootstrap.cc:492] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:42.177630 983 leader_election.cc:343] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 3eeb085e764941198ae7f653bc24757d
I20250112 20:51:42.177614 1163 ts_tablet_manager.cc:1397] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20250112 20:51:42.178499 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:42.178659 983 leader_election.cc:343] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 3eeb085e764941198ae7f653bc24757d
I20250112 20:51:42.178802 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.178802 983 leader_election.cc:304] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.179144 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.179309 770 raft_consensus.cc:3054] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.179437 2484 raft_consensus.cc:2743] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.181980 770 raft_consensus.cc:2463] T 2111e988ea634e999a1bbbf7b270153b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.182264 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.182281 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fec3d1a320fe450dbab6cf50e1380d8d" } modify_external_catalogs: true
I20250112 20:51:42.182435 774 raft_consensus.cc:2463] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.182368 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a58bef1719324f5c87d4e91573f71d90" } new_table_name: "default.Table_d8ca8716dceb4d39875ded505a45fc1a" modify_external_catalogs: true
I20250112 20:51:42.182474 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fec3d1a320fe450dbab6cf50e1380d8d" } modify_external_catalogs: true
I20250112 20:51:42.186313 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e6d67503da2e4fd9aedb3c00d4c8e4cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:42.186574 2484 raft_consensus.cc:491] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.186694 2484 raft_consensus.cc:513] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:42.187119 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6d67503da2e4fd9aedb3c00d4c8e4cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.187229 2484 leader_election.cc:290] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:42.188184 983 leader_election.cc:343] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9c6d5d425e6a493daa46900cbd8db166
W20250112 20:51:42.188349 983 leader_election.cc:343] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:42.188409 983 leader_election.cc:304] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.188691 2484 raft_consensus.cc:2743] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.178674 1163 raft_consensus.cc:357] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.188827 1163 raft_consensus.cc:383] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.188874 1163 raft_consensus.cc:738] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.189091 1163 consensus_queue.cc:260] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.190361 1163 ts_tablet_manager.cc:1428] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:42.190670 1163 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.194550 1163 tablet_bootstrap.cc:654] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:42.199195 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60408b7819484605a95cede93f38a9d5 consensus state has no leader
W20250112 20:51:42.202591 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf6d73ab981c4e87aa229a49fb3b6913 consensus state has no leader
W20250112 20:51:42.179262 847 leader_election.cc:343] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.182930 1178 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.188093 2415 raft_consensus.cc:491] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.203078 2415 raft_consensus.cc:513] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:42.203166 847 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.203648 2415 leader_election.cc:290] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.204197 847 leader_election.cc:304] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.204375 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f34aa32d6474561988b2cf3d0e8fd3b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.204586 847 leader_election.cc:304] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.205149 2415 raft_consensus.cc:2798] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.205150 2520 raft_consensus.cc:2798] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.205286 2415 raft_consensus.cc:491] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.205339 2520 raft_consensus.cc:695] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.205472 2415 raft_consensus.cc:3054] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:42.205893 847 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.205825 2520 consensus_queue.cc:237] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:42.206353 847 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:42.206449 847 leader_election.cc:304] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.206960 2521 raft_consensus.cc:2743] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.208367 2415 raft_consensus.cc:513] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.214406 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.214099 2415 leader_election.cc:290] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:42.214799 847 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.215169 1178 ts_tablet_manager.cc:1397] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20250112 20:51:42.216282 1178 raft_consensus.cc:357] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.216576 1178 raft_consensus.cc:383] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.216711 1178 raft_consensus.cc:738] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.216784 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b788acf8047acb34ca2596bbf141b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.217018 774 raft_consensus.cc:3054] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.217136 1178 consensus_queue.cc:260] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.217890 1178 ts_tablet_manager.cc:1428] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:42.218320 1178 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.219485 774 raft_consensus.cc:2463] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.219946 847 leader_election.cc:304] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.220274 2108 raft_consensus.cc:2798] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.220592 2108 raft_consensus.cc:695] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.220712 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d29316b9d2441bca97f56bdb0e56f0d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:42.220992 2108 consensus_queue.cc:237] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.223829 1178 tablet_bootstrap.cc:654] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.231035 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a06a212d24b846debb4ec624d3a1f24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.231738 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a06a212d24b846debb4ec624d3a1f24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.232957 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ce095329e25c4874a1e151a4f5d84c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.233546 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce095329e25c4874a1e151a4f5d84c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.234357 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 5f264d84ae4846d1afaaabc41d5789f4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:42.240248 1178 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.241324 1178 ts_tablet_manager.cc:1397] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20250112 20:51:42.244127 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fc795f0197f74154a56a81c67c43185c" } modify_external_catalogs: true
I20250112 20:51:42.246117 1178 raft_consensus.cc:357] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.246789 1178 raft_consensus.cc:383] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.247149 1178 raft_consensus.cc:738] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.248279 1178 consensus_queue.cc:260] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.250458 1178 ts_tablet_manager.cc:1428] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250112 20:51:42.251794 1178 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.253309 2484 raft_consensus.cc:491] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.253456 2484 raft_consensus.cc:513] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.254036 2484 leader_election.cc:290] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:42.254948 983 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 5d384628ba9b4f81b768eda4f796632b
W20250112 20:51:42.256206 983 leader_election.cc:343] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:42.256310 983 leader_election.cc:304] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.256692 2484 raft_consensus.cc:2743] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.262923 1178 tablet_bootstrap.cc:654] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.292418 1178 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.292753 1178 ts_tablet_manager.cc:1397] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.005s
I20250112 20:51:42.294019 1178 raft_consensus.cc:357] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.294404 1178 raft_consensus.cc:383] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.294530 1178 raft_consensus.cc:738] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.294875 1178 consensus_queue.cc:260] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.295684 1178 ts_tablet_manager.cc:1428] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:42.296720 1178 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.305691 1178 tablet_bootstrap.cc:654] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.317152 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c4fafabe80bf4a93a9c354c73ad78f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.317919 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4fafabe80bf4a93a9c354c73ad78f81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.319440 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ec3a27d13db349b4a78c058fdea0ad2d" } new_table_name: "default.Table_89f62bae93b94b418c7f625cb56de125" modify_external_catalogs: true
I20250112 20:51:42.320299 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_28283ef30e624ee992db4e187bd92df0" } new_table_name: "default.Table_1e8f92a0b14c40829a4e275e8327e838" modify_external_catalogs: true
I20250112 20:51:42.322744 1178 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.323082 1178 ts_tablet_manager.cc:1397] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
I20250112 20:51:42.324643 1178 raft_consensus.cc:357] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.324926 1178 raft_consensus.cc:383] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.325037 1178 raft_consensus.cc:738] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.325409 1178 consensus_queue.cc:260] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.326169 1178 ts_tablet_manager.cc:1428] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:42.327323 1178 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.334448 2108 raft_consensus.cc:491] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.334580 2108 raft_consensus.cc:513] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.335165 2108 leader_election.cc:290] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.335824 2484 raft_consensus.cc:491] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.335968 2484 raft_consensus.cc:513] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.336537 2484 leader_election.cc:290] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.336746 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.337698 847 leader_election.cc:343] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:42.338392 983 leader_election.cc:343] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4bd46a422093495ba8b5b03039874a32
I20250112 20:51:42.340219 1178 tablet_bootstrap.cc:654] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.342674 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:42.342804 983 leader_election.cc:343] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4bd46a422093495ba8b5b03039874a32
I20250112 20:51:42.342897 983 leader_election.cc:304] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.343245 2484 raft_consensus.cc:2743] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:42.343572 847 leader_election.cc:343] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.343696 847 leader_election.cc:304] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.344094 2108 raft_consensus.cc:2743] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.350899 1044 raft_consensus.cc:3054] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.354226 2108 consensus_queue.cc:1035] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:42.364907 1178 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.365592 1178 ts_tablet_manager.cc:1397] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20250112 20:51:42.368973 1178 raft_consensus.cc:357] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.375298 1178 raft_consensus.cc:383] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.376540 1178 raft_consensus.cc:738] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.376832 1163 tablet_bootstrap.cc:492] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.377876 1163 ts_tablet_manager.cc:1397] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.187s user 0.029s sys 0.022s
I20250112 20:51:42.377864 1178 consensus_queue.cc:260] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.379806 1164 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.380190 1164 ts_tablet_manager.cc:1397] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.407s user 0.048s sys 0.123s
I20250112 20:51:42.381922 1164 raft_consensus.cc:357] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.382314 1164 raft_consensus.cc:383] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.382403 1164 raft_consensus.cc:738] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.382779 1164 consensus_queue.cc:260] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.384303 1178 ts_tablet_manager.cc:1428] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 20:51:42.384903 1164 ts_tablet_manager.cc:1428] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:42.386565 1178 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.386574 1164 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.387323 2484 raft_consensus.cc:491] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.387437 2484 raft_consensus.cc:513] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.387982 2484 leader_election.cc:290] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.387096 1163 raft_consensus.cc:357] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.388326 1163 raft_consensus.cc:383] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.388749 1163 raft_consensus.cc:738] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:42.389600 983 leader_election.cc:343] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 64887b3578284cb286ee122ab8c626c4
I20250112 20:51:42.390071 1163 consensus_queue.cc:260] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:42.393962 983 leader_election.cc:343] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 64887b3578284cb286ee122ab8c626c4
I20250112 20:51:42.394110 983 leader_election.cc:304] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.393967 1163 ts_tablet_manager.cc:1428] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250112 20:51:42.394510 2484 raft_consensus.cc:2743] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.395476 1163 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.398249 1164 tablet_bootstrap.cc:654] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.403950 1178 tablet_bootstrap.cc:654] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.405341 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f9055531da4c453fa722eb7c78fa0e78
I20250112 20:51:42.405892 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fc90d10b52794ebea4e3e19db3cc8dc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:42.406055 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eaed25edb5a043188e95b48cdd144890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.406497 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc90d10b52794ebea4e3e19db3cc8dc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:42.406843 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaed25edb5a043188e95b48cdd144890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.408044 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb with delete_type TABLET_DATA_DELETED (replaced by tablet 23dc8a695ccd4f88bd108304add01a4e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.408449 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 942e8f74c4334c0eb4b441c020605631 (DEFAULT_TABLE table=default.Table_84c10c7cd6164c07aed6ccf21fe30476 [id=10a0c5cdad9d4deb98cca37ae56d1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.408915 2447 tablet_replica.cc:331] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:42.409050 892 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.409132 2447 raft_consensus.cc:2238] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.409253 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb with delete_type TABLET_DATA_DELETED (replaced by tablet 23dc8a695ccd4f88bd108304add01a4e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.409291 2447 raft_consensus.cc:2267] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.409479 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.408931 1163 tablet_bootstrap.cc:654] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.409983 2447 ts_tablet_manager.cc:1905] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.408900 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942e8f74c4334c0eb4b441c020605631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.413426 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb with delete_type TABLET_DATA_DELETED (replaced by tablet 23dc8a695ccd4f88bd108304add01a4e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.417200 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ca52e028a3b84bfe8dcb38464702a6ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:42.417649 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca52e028a3b84bfe8dcb38464702a6ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:42.420230 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a20d85a4be874b0b90c528a631a29aed consensus state has no leader
I20250112 20:51:42.421452 2447 ts_tablet_manager.cc:1918] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.421535 2447 log.cc:1198] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:42.421886 2447 ts_tablet_manager.cc:1939] T 64e1c32cab2c4b438d3ff4a7e674feeb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:42.423009 1164 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.423168 1164 ts_tablet_manager.cc:1397] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20250112 20:51:42.423445 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93" } modify_external_catalogs: true
I20250112 20:51:42.423540 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1c840ad1fe9c49aba52e7a3dc7281fe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:42.423691 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_557dc1e9ed3d4bad8dc2487fa9e2bb93" } modify_external_catalogs: true
W20250112 20:51:42.424003 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c840ad1fe9c49aba52e7a3dc7281fe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:42.425287 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_93a664f616254b98a1914cedad79b639" } modify_external_catalogs: true
I20250112 20:51:42.425475 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_93a664f616254b98a1914cedad79b639" } modify_external_catalogs: true
I20250112 20:51:42.425726 1164 raft_consensus.cc:357] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.425932 1164 raft_consensus.cc:383] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.425993 1164 raft_consensus.cc:738] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.426214 1164 consensus_queue.cc:260] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.426692 1164 ts_tablet_manager.cc:1428] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:42.426950 1164 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.430440 883 tablet_service.cc:1467] Processing CreateTablet for tablet 942e8f74c4334c0eb4b441c020605631 (DEFAULT_TABLE table=default.Table_84c10c7cd6164c07aed6ccf21fe30476 [id=10a0c5cdad9d4deb98cca37ae56d1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.430805 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942e8f74c4334c0eb4b441c020605631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.432286 1164 tablet_bootstrap.cc:654] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.433100 747 tablet_service.cc:1467] Processing CreateTablet for tablet 942e8f74c4334c0eb4b441c020605631 (DEFAULT_TABLE table=default.Table_84c10c7cd6164c07aed6ccf21fe30476 [id=10a0c5cdad9d4deb98cca37ae56d1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.433413 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942e8f74c4334c0eb4b441c020605631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.435967 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fc795f0197f74154a56a81c67c43185c" } modify_external_catalogs: true
I20250112 20:51:42.436379 1164 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.436484 1164 ts_tablet_manager.cc:1397] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250112 20:51:42.437199 1164 raft_consensus.cc:357] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.437371 1164 raft_consensus.cc:383] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.437426 1164 raft_consensus.cc:738] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.437623 1164 consensus_queue.cc:260] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.438103 1164 ts_tablet_manager.cc:1428] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:42.438377 1164 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.438510 1186 raft_consensus.cc:2238] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.438678 1186 raft_consensus.cc:2267] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.439170 1186 ts_tablet_manager.cc:1905] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.439883 1164 tablet_bootstrap.cc:654] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.441138 2415 raft_consensus.cc:491] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.441267 2415 raft_consensus.cc:513] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.441788 2415 leader_election.cc:290] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.444938 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f1119a719b4c0fa47650dc1fb1c27d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.445482 847 leader_election.cc:343] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.445619 1186 ts_tablet_manager.cc:1918] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.445775 1186 log.cc:1198] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f720bd9710d2476d9632699a2732cb05
I20250112 20:51:42.446266 1186 ts_tablet_manager.cc:1939] T f720bd9710d2476d9632699a2732cb05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:42.446656 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 774) took 1153 ms. Trace:
I20250112 20:51:42.446808 1186 rpcz_store.cc:276] 0112 20:51:41.292848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.292901 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:42.163828 (+870927us) ts_tablet_manager.cc:1145] Deleting tablet f720bd9710d2476d9632699a2732cb05
0112 20:51:42.163834 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.442043 (+278209us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.445549 (+ 3506us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.446637 (+ 1088us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":274387,"tablet-delete.queue_time_us":863647}
I20250112 20:51:42.447381 1186 tablet_replica.cc:331] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:42.447677 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f1119a719b4c0fa47650dc1fb1c27d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.447909 772 raft_consensus.cc:2463] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.448258 1186 raft_consensus.cc:2238] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.448328 847 leader_election.cc:304] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.448457 1186 raft_consensus.cc:2267] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.448722 2415 raft_consensus.cc:2798] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.448871 2415 raft_consensus.cc:491] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.448957 2415 raft_consensus.cc:3054] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.451154 2415 raft_consensus.cc:513] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.452220 1164 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.455156 1164 ts_tablet_manager.cc:1397] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250112 20:51:42.455883 1164 raft_consensus.cc:357] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.456061 1164 raft_consensus.cc:383] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.456132 1164 raft_consensus.cc:738] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.456352 2415 leader_election.cc:290] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.456377 1164 consensus_queue.cc:260] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.456854 1164 ts_tablet_manager.cc:1428] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:42.457021 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f1119a719b4c0fa47650dc1fb1c27d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.457113 1164 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.457276 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f1119a719b4c0fa47650dc1fb1c27d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.457494 772 raft_consensus.cc:3054] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.458593 1164 tablet_bootstrap.cc:654] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.448962 1186 ts_tablet_manager.cc:1905] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.460922 772 raft_consensus.cc:2463] T 64f1119a719b4c0fa47650dc1fb1c27d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.461295 847 leader_election.cc:304] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:42.461685 847 leader_election.cc:343] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.461980 2415 raft_consensus.cc:2798] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.462106 2415 raft_consensus.cc:695] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.462370 2415 consensus_queue.cc:237] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.463565 2208 raft_consensus.cc:491] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.463707 2208 raft_consensus.cc:513] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.464419 2208 leader_election.cc:290] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:42.465098 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092a31de9bc47e4bae5f5aebb180c6d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:42.465173 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092a31de9bc47e4bae5f5aebb180c6d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.465324 913 raft_consensus.cc:2463] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:42.465740 710 leader_election.cc:304] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:42.466020 2208 raft_consensus.cc:2798] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.466100 2208 raft_consensus.cc:491] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.466176 2208 raft_consensus.cc:3054] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:42.468981 710 leader_election.cc:343] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.469990 2208 raft_consensus.cc:513] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.470741 2208 leader_election.cc:290] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:42.471932 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 0a87da1ea4054ffc81b9c1c792f0cc5d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:42.472934 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092a31de9bc47e4bae5f5aebb180c6d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.473701 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092a31de9bc47e4bae5f5aebb180c6d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:42.473939 913 raft_consensus.cc:3054] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:42.474457 710 leader_election.cc:343] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.477401 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 261fe76c57b14b35957ca43606cfc02f (DEFAULT_TABLE table=default.Table_d47332a374d74462aa73b72ac64269bf [id=ea5d9349744440cf864357fb825f70d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.477782 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261fe76c57b14b35957ca43606cfc02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.476701 913 raft_consensus.cc:2463] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:42.478633 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet fae080f8e4b7436b9166c85604a77dfa
I20250112 20:51:42.481916 1014 tablet_service.cc:1467] Processing CreateTablet for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb (DEFAULT_TABLE table=default.Table_60a304553d7548dc89dd2d4b1e5f4bd3 [id=d02e9ac608b24d118be25725ea308187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.482266 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.484604 1164 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.484746 1164 ts_tablet_manager.cc:1397] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20250112 20:51:42.485558 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 766) took 1253 ms. Trace:
I20250112 20:51:42.485666 1185 rpcz_store.cc:276] 0112 20:51:41.231690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.233654 (+ 1964us) service_pool.cc:225] Handling call
0112 20:51:41.889332 (+655678us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:41.889340 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:41.905530 (+ 16190us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:41.971505 (+ 65975us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.485537 (+514032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":655543}
I20250112 20:51:42.486074 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 767) took 1254 ms. Trace:
I20250112 20:51:42.486163 1185 rpcz_store.cc:276] 0112 20:51:41.231785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.236227 (+ 4442us) service_pool.cc:225] Handling call
0112 20:51:42.485980 (+1249753us) ts_tablet_manager.cc:1145] Deleting tablet 35dcd1a120684446bbe19303fb8416ef
0112 20:51:42.485990 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.486060 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249614}
I20250112 20:51:42.486454 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 769) took 1236 ms. Trace:
I20250112 20:51:42.486527 1185 rpcz_store.cc:276] 0112 20:51:41.249520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.265195 (+ 15675us) service_pool.cc:225] Handling call
0112 20:51:42.486372 (+1221177us) ts_tablet_manager.cc:1145] Deleting tablet 762388f687dd4b05ae6ffdaf8c64c999
0112 20:51:42.486379 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.486443 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1220982}
I20250112 20:51:42.486734 1185 tablet_replica.cc:331] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:42.486883 1185 raft_consensus.cc:2238] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.487013 1185 raft_consensus.cc:2267] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.487486 1185 ts_tablet_manager.cc:1905] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.488592 747 tablet_service.cc:1467] Processing CreateTablet for tablet 261fe76c57b14b35957ca43606cfc02f (DEFAULT_TABLE table=default.Table_d47332a374d74462aa73b72ac64269bf [id=ea5d9349744440cf864357fb825f70d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.488973 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261fe76c57b14b35957ca43606cfc02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.485556 1164 raft_consensus.cc:357] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.491149 1164 raft_consensus.cc:383] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.491209 1164 raft_consensus.cc:738] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.491402 1164 consensus_queue.cc:260] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.493176 883 tablet_service.cc:1467] Processing CreateTablet for tablet 261fe76c57b14b35957ca43606cfc02f (DEFAULT_TABLE table=default.Table_d47332a374d74462aa73b72ac64269bf [id=ea5d9349744440cf864357fb825f70d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.493366 1164 ts_tablet_manager.cc:1428] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250112 20:51:42.493635 1164 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.493656 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261fe76c57b14b35957ca43606cfc02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.495144 1164 tablet_bootstrap.cc:654] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.496490 2484 raft_consensus.cc:491] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.496611 2484 raft_consensus.cc:513] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:42.497390 2484 leader_election.cc:290] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:42.498492 983 leader_election.cc:343] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d0829e9dd5854961b5402b5710624612
I20250112 20:51:42.499665 710 leader_election.cc:304] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.500135 2059 raft_consensus.cc:2798] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.500299 2059 raft_consensus.cc:695] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
W20250112 20:51:42.500346 983 leader_election.cc:343] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d0829e9dd5854961b5402b5710624612
I20250112 20:51:42.500451 983 leader_election.cc:304] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.500569 2059 consensus_queue.cc:237] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.500802 2484 raft_consensus.cc:2743] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:42.503373 847 consensus_peers.cc:487] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:42.504308 741 tablet_service.cc:1467] Processing CreateTablet for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb (DEFAULT_TABLE table=default.Table_60a304553d7548dc89dd2d4b1e5f4bd3 [id=d02e9ac608b24d118be25725ea308187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.504632 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.509374 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5f264d84ae4846d1afaaabc41d5789f4
I20250112 20:51:42.509920 453 catalog_manager.cc:6596] ReplaceTablet: tablet 5f264d84ae4846d1afaaabc41d5789f4 deleted and replaced by tablet f424288fe62849edbbd2d38e89331fb7
I20250112 20:51:42.511286 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f" } modify_external_catalogs: true
I20250112 20:51:42.511466 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f" } modify_external_catalogs: true
I20250112 20:51:42.511812 1163 tablet_bootstrap.cc:492] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.511945 1163 ts_tablet_manager.cc:1397] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.117s user 0.003s sys 0.021s
I20250112 20:51:42.512682 1163 raft_consensus.cc:357] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.512895 1163 raft_consensus.cc:383] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.512954 1163 raft_consensus.cc:738] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.521219 881 tablet_service.cc:1467] Processing CreateTablet for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb (DEFAULT_TABLE table=default.Table_60a304553d7548dc89dd2d4b1e5f4bd3 [id=d02e9ac608b24d118be25725ea308187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.521303 1185 ts_tablet_manager.cc:1918] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:42.521353 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 36f95046c87846b2aa7e15430c1480b5
I20250112 20:51:42.521411 1185 log.cc:1198] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8b7b788acf8047acb34ca2596bbf141b
I20250112 20:51:42.521617 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28b2e5571f040c2a9a51ea1ea5f3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.521685 1185 ts_tablet_manager.cc:1939] T 8b7b788acf8047acb34ca2596bbf141b P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:42.521647 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8cd06d785b494db477f87543b9bd07 (DEFAULT_TABLE table=default.Table_c331f1ff143e43c99fac735c4547b362 [id=41a0e106324549b69128a9fc7ba34d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.522193 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8cd06d785b494db477f87543b9bd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.522468 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 770) took 1265 ms. Trace:
I20250112 20:51:42.522558 1185 rpcz_store.cc:276] 0112 20:51:41.256880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.267106 (+ 10226us) service_pool.cc:225] Handling call
0112 20:51:42.486702 (+1219596us) ts_tablet_manager.cc:1145] Deleting tablet 8b7b788acf8047acb34ca2596bbf141b
0112 20:51:42.486709 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.514318 (+ 27609us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.521207 (+ 6889us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.522449 (+ 1242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219427}
I20250112 20:51:42.525769 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2440e0bdf8ad4ab7805c892e28834acd
I20250112 20:51:42.528170 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8cd06d785b494db477f87543b9bd07 (DEFAULT_TABLE table=default.Table_c331f1ff143e43c99fac735c4547b362 [id=41a0e106324549b69128a9fc7ba34d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.529073 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8cd06d785b494db477f87543b9bd07 (DEFAULT_TABLE table=default.Table_c331f1ff143e43c99fac735c4547b362 [id=41a0e106324549b69128a9fc7ba34d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.529412 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8cd06d785b494db477f87543b9bd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.529330 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8c697bba271f4455956b9a2511155b79
I20250112 20:51:42.529726 1163 consensus_queue.cc:260] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.530272 1163 ts_tablet_manager.cc:1428] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250112 20:51:42.531575 1163 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.532069 2059 raft_consensus.cc:491] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.532194 2059 raft_consensus.cc:513] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.532765 2059 leader_election.cc:290] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:42.533365 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 75790518579d496fb0efb12b497fcc75 (DEFAULT_TABLE table=default.Table_1dff4cb9aa364c87b97f2d05462b70ac [id=c855ae736f0d402e8fb021a124771fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.533708 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75790518579d496fb0efb12b497fcc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.533857 744 tablet_service.cc:1467] Processing CreateTablet for tablet 75790518579d496fb0efb12b497fcc75 (DEFAULT_TABLE table=default.Table_1dff4cb9aa364c87b97f2d05462b70ac [id=c855ae736f0d402e8fb021a124771fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.534188 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9219fc971c4d15b0d383b20b61e540" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:42.534371 913 raft_consensus.cc:2463] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:42.535241 710 leader_election.cc:304] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:42.535467 2059 raft_consensus.cc:2798] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.535532 2059 raft_consensus.cc:491] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.535588 2059 raft_consensus.cc:3054] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.536515 892 tablet_service.cc:1467] Processing CreateTablet for tablet 75790518579d496fb0efb12b497fcc75 (DEFAULT_TABLE table=default.Table_1dff4cb9aa364c87b97f2d05462b70ac [id=c855ae736f0d402e8fb021a124771fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.536844 885 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.536911 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9219fc971c4d15b0d383b20b61e540" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.537405 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75790518579d496fb0efb12b497fcc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.538257 2059 raft_consensus.cc:513] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.538769 2059 leader_election.cc:290] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:42.539319 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9219fc971c4d15b0d383b20b61e540" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:42.539474 913 raft_consensus.cc:3054] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.541149 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9219fc971c4d15b0d383b20b61e540" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:42.541649 710 leader_election.cc:343] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.541888 913 raft_consensus.cc:2463] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:42.541949 710 leader_election.cc:343] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.543146 1163 tablet_bootstrap.cc:654] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.544008 710 leader_election.cc:304] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.544351 2059 raft_consensus.cc:2798] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.544449 2059 raft_consensus.cc:695] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:42.544656 2059 consensus_queue.cc:237] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.547430 1185 tablet_replica.cc:331] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:42.547816 1185 raft_consensus.cc:2238] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:42.548141 749 tablet_service.cc:1467] Processing CreateTablet for tablet 23dc8a695ccd4f88bd108304add01a4e (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.552881 1163 tablet_bootstrap.cc:492] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.548632 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75790518579d496fb0efb12b497fcc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.553315 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8cd06d785b494db477f87543b9bd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.553515 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc8a695ccd4f88bd108304add01a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.553858 1185 raft_consensus.cc:2267] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.554647 1185 ts_tablet_manager.cc:1905] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.553145 1163 ts_tablet_manager.cc:1397] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20250112 20:51:42.560554 1163 raft_consensus.cc:357] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.560881 1163 raft_consensus.cc:383] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.560962 1163 raft_consensus.cc:738] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.561209 1163 consensus_queue.cc:260] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.561820 1163 ts_tablet_manager.cc:1428] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:42.562104 1163 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.571940 1185 ts_tablet_manager.cc:1918] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.572561 1163 tablet_bootstrap.cc:654] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.573293 1185 log.cc:1198] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f720bd9710d2476d9632699a2732cb05
I20250112 20:51:42.573829 1185 ts_tablet_manager.cc:1939] T f720bd9710d2476d9632699a2732cb05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:42.574965 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 772) took 1290 ms. Trace:
I20250112 20:51:42.576277 881 tablet_service.cc:1467] Processing CreateTablet for tablet 23dc8a695ccd4f88bd108304add01a4e (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.577991 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 23dc8a695ccd4f88bd108304add01a4e (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.578176 2538 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b04ae76622ac46f5a584a582e372bc05)
I20250112 20:51:42.578393 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc8a695ccd4f88bd108304add01a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.578367 2538 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } }
I20250112 20:51:42.579483 2538 leader_election.cc:290] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b04ae76622ac46f5a584a582e372bc05 (127.0.104.62:35199), b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271)
I20250112 20:51:42.579679 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b04ae76622ac46f5a584a582e372bc05" is_pre_election: true
I20250112 20:51:42.581940 1352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5c09e899184de6b770dca873501062" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: false dest_uuid: "b985fd656ce44131bc91ccfeb79587c5" is_pre_election: true
I20250112 20:51:42.582752 2032 leader_election.cc:304] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f5c09e899184de6b770dca873501062; no voters: b04ae76622ac46f5a584a582e372bc05, b985fd656ce44131bc91ccfeb79587c5
I20250112 20:51:42.586735 2538 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f5c09e899184de6b770dca873501062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:42.586582 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f3ff8ac6f4d94c10b03a764bbf910c44 (table default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]) successfully deleted
I20250112 20:51:42.587414 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3bbb679585fc44699553fe85ea044566 (table default.Table_8eeab516a6794571a8336068218e7a3a [id=2bc2a6f29dda4e13a781af77ba66b46f]) successfully deleted
W20250112 20:51:42.587644 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 43d341f6ac6848d9a8e60f689dd24a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43d341f6ac6848d9a8e60f689dd24a66 already in progress: creating tablet
I20250112 20:51:42.587445 1185 rpcz_store.cc:276] 0112 20:51:41.284571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.285442 (+ 871us) service_pool.cc:225] Handling call
0112 20:51:42.522849 (+1237407us) ts_tablet_manager.cc:1145] Deleting tablet f720bd9710d2476d9632699a2732cb05
0112 20:51:42.547346 (+ 24497us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.562936 (+ 15590us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.571855 (+ 8919us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.574941 (+ 3086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237129}
W20250112 20:51:42.587908 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e2c41c2c92b442b986d6b75708c7420c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c41c2c92b442b986d6b75708c7420c already in progress: creating tablet
I20250112 20:51:42.588104 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d0829e9dd5854961b5402b5710624612 (table default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
W20250112 20:51:42.588347 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9a23281d9b6f47a5bf627374fa0cfbf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a23281d9b6f47a5bf627374fa0cfbf2 already in progress: creating tablet
I20250112 20:51:42.588537 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]) successfully deleted
I20250112 20:51:42.588912 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 782) took 1168 ms. Trace:
I20250112 20:51:42.589205 1185 rpcz_store.cc:276] 0112 20:51:41.420311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.420879 (+ 568us) service_pool.cc:225] Handling call
0112 20:51:42.588565 (+1167686us) ts_tablet_manager.cc:1145] Deleting tablet 1a505986bc6146829cbafd0de6682d93
0112 20:51:42.588576 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.588892 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167488}
I20250112 20:51:42.589361 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b310e4135c2b424fb006a38a60e41066 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
W20250112 20:51:42.589583 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8f6995f7fbbf480995353b66749fbc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f6995f7fbbf480995353b66749fbc10 already in progress: creating tablet
W20250112 20:51:42.589797 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8b7b788acf8047acb34ca2596bbf141b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b7b788acf8047acb34ca2596bbf141b already in progress: creating tablet
I20250112 20:51:42.589998 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]) successfully deleted
W20250112 20:51:42.590255 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 762388f687dd4b05ae6ffdaf8c64c999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 762388f687dd4b05ae6ffdaf8c64c999 already in progress: creating tablet
I20250112 20:51:42.590277 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 784) took 1169 ms. Trace:
W20250112 20:51:42.590473 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cbdc5c117ca9437091a1667642c7c846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdc5c117ca9437091a1667642c7c846 already in progress: creating tablet
W20250112 20:51:42.590679 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1a505986bc6146829cbafd0de6682d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a505986bc6146829cbafd0de6682d93 already in progress: creating tablet
I20250112 20:51:42.590704 1163 tablet_bootstrap.cc:492] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.590879 1163 ts_tablet_manager.cc:1397] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
W20250112 20:51:42.590939 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 49bfc4fbe78745049699c8383c724f38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bfc4fbe78745049699c8383c724f38 already in progress: creating tablet
I20250112 20:51:42.590673 1185 rpcz_store.cc:276] 0112 20:51:41.420500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.421229 (+ 729us) service_pool.cc:225] Handling call
0112 20:51:42.589997 (+1168768us) ts_tablet_manager.cc:1145] Deleting tablet 49bfc4fbe78745049699c8383c724f38
0112 20:51:42.590008 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.590260 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168538}
W20250112 20:51:42.591171 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ab9be1f065ec4605a0a3fe69770793d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab9be1f065ec4605a0a3fe69770793d0 already in progress: creating tablet
I20250112 20:51:42.591351 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4ce4f31cf8f24e35b7281d4a8b5667a0 (table default.Table_c04acf4e49804989ad6fbb05c89a8f20 [id=24707a281ea447f99afb3b99bccb69f1]) successfully deleted
W20250112 20:51:42.591562 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b519b0dc51a240629b6376767fc19e5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b519b0dc51a240629b6376767fc19e5d already in progress: creating tablet
W20250112 20:51:42.591775 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d3ea347fd2694016a3cd3721b0cab608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3ea347fd2694016a3cd3721b0cab608 already in progress: creating tablet
I20250112 20:51:42.591830 1163 raft_consensus.cc:357] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:42.592000 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 4633e6870ccd4ce5b2458bb22155ba42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4633e6870ccd4ce5b2458bb22155ba42 already in progress: creating tablet
I20250112 20:51:42.592073 1163 raft_consensus.cc:383] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.592144 1163 raft_consensus.cc:738] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:42.592186 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b37e9e95c6b745ccaba5446b04abcc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37e9e95c6b745ccaba5446b04abcc9d already in progress: creating tablet
I20250112 20:51:42.592360 1163 consensus_queue.cc:260] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.592514 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]) successfully deleted
I20250112 20:51:42.592834 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 783) took 1172 ms. Trace:
I20250112 20:51:42.592989 1163 ts_tablet_manager.cc:1428] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:42.593194 1185 rpcz_store.cc:276] 0112 20:51:41.420397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.449811 (+ 29414us) service_pool.cc:225] Handling call
0112 20:51:42.591981 (+1142170us) ts_tablet_manager.cc:1145] Deleting tablet cbdc5c117ca9437091a1667642c7c846
0112 20:51:42.591992 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.592818 (+ 826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141990}
I20250112 20:51:42.594447 1185 tablet_replica.cc:331] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:42.594833 1185 raft_consensus.cc:2238] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:42.596369 1185 pending_rounds.cc:62] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:42.597810 1178 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.598105 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23dc8a695ccd4f88bd108304add01a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.598125 1178 ts_tablet_manager.cc:1397] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.212s user 0.021s sys 0.009s
I20250112 20:51:42.600440 1178 raft_consensus.cc:357] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.600620 1178 raft_consensus.cc:383] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.600665 1178 raft_consensus.cc:738] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.600996 1178 consensus_queue.cc:260] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.601513 1178 ts_tablet_manager.cc:1428] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:42.601868 1178 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.603487 1178 tablet_bootstrap.cc:654] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.596693 1185 pending_rounds.cc:69] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585057159139328 op_type: NO_OP noop_request { }
I20250112 20:51:42.603610 1185 raft_consensus.cc:2883] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:42.603719 1185 raft_consensus.cc:2267] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.605532 1185 ts_tablet_manager.cc:1905] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.613404 1164 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.613540 1164 ts_tablet_manager.cc:1397] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.019s
I20250112 20:51:42.613801 2415 raft_consensus.cc:491] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.607648 1186 ts_tablet_manager.cc:1918] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:42.613950 1186 log.cc:1198] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:42.613921 2415 raft_consensus.cc:513] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.614326 1186 ts_tablet_manager.cc:1939] T 267ebb3c30194d9fa6a092d9bec4eadf P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:42.614286 1164 raft_consensus.cc:357] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.614450 1164 raft_consensus.cc:383] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.614483 2415 leader_election.cc:290] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.614501 1164 raft_consensus.cc:738] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.614638 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 771) took 1322 ms. Trace:
I20250112 20:51:42.615096 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ab0ae10f09469fa4f913e049002481" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.615121 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ab0ae10f09469fa4f913e049002481" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:42.615252 774 raft_consensus.cc:2463] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.615577 847 leader_election.cc:304] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:42.615723 1186 rpcz_store.cc:276] 0112 20:51:41.292624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.292945 (+ 321us) service_pool.cc:225] Handling call
0112 20:51:42.447337 (+1154392us) ts_tablet_manager.cc:1145] Deleting tablet 267ebb3c30194d9fa6a092d9bec4eadf
0112 20:51:42.447347 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.598029 (+150682us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.607558 (+ 9529us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.614621 (+ 7063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146206}
W20250112 20:51:42.615916 847 leader_election.cc:343] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.616268 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 770) took 1345 ms. Trace:
I20250112 20:51:42.614709 1164 consensus_queue.cc:260] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.616422 2415 raft_consensus.cc:2798] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.616623 2415 raft_consensus.cc:491] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.616704 2415 raft_consensus.cc:3054] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.618798 1164 ts_tablet_manager.cc:1428] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:42.619058 1164 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.619091 2371 raft_consensus.cc:491] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.619235 2371 raft_consensus.cc:513] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.619937 2371 leader_election.cc:290] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.620266 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:42.620725 847 leader_election.cc:343] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.621516 2415 raft_consensus.cc:513] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.621743 1164 tablet_bootstrap.cc:654] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.621979 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.622455 847 leader_election.cc:343] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.622552 847 leader_election.cc:304] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.622757 2415 leader_election.cc:290] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.623162 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ab0ae10f09469fa4f913e049002481" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.623267 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ab0ae10f09469fa4f913e049002481" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.622885 2415 raft_consensus.cc:2743] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:42.623447 774 raft_consensus.cc:3054] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.624279 1186 rpcz_store.cc:276] 0112 20:51:41.292492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.293098 (+ 606us) service_pool.cc:225] Handling call
0112 20:51:42.616120 (+1323022us) ts_tablet_manager.cc:1145] Deleting tablet 762388f687dd4b05ae6ffdaf8c64c999
0112 20:51:42.616134 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.616251 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314406}
I20250112 20:51:42.624568 1186 tablet_replica.cc:331] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:42.624737 1186 raft_consensus.cc:2238] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.624879 1186 raft_consensus.cc:2267] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.626338 774 raft_consensus.cc:2463] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:42.625191 847 leader_election.cc:343] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.626740 847 leader_election.cc:304] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.626971 1186 ts_tablet_manager.cc:1905] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.627183 2371 raft_consensus.cc:2798] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.627290 2371 raft_consensus.cc:695] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.627502 2371 consensus_queue.cc:237] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.629657 1185 ts_tablet_manager.cc:1918] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.629755 1185 log.cc:1198] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/70653110aec74727acec9d63c2eb0f05
I20250112 20:51:42.630092 1185 ts_tablet_manager.cc:1939] T 70653110aec74727acec9d63c2eb0f05 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:42.632062 1178 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.632253 1178 ts_tablet_manager.cc:1397] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20250112 20:51:42.632993 1178 raft_consensus.cc:357] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.633260 1178 raft_consensus.cc:383] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.633448 1178 raft_consensus.cc:738] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.633716 1178 consensus_queue.cc:260] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.634279 1178 ts_tablet_manager.cc:1428] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:42.634670 1178 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.636363 1163 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.640966 1178 tablet_bootstrap.cc:654] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.643429 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 785) took 1064 ms. Trace:
I20250112 20:51:42.643759 1163 tablet_bootstrap.cc:654] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.648300 1185 rpcz_store.cc:276] 0112 20:51:41.578810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.578885 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:42.594402 (+1015517us) ts_tablet_manager.cc:1145] Deleting tablet 70653110aec74727acec9d63c2eb0f05
0112 20:51:42.594411 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.608475 (+ 14064us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.629563 (+ 21088us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.643407 (+ 13844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015191}
I20250112 20:51:42.649204 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 786) took 1066 ms. Trace:
I20250112 20:51:42.650637 1185 rpcz_store.cc:276] 0112 20:51:41.582790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.585150 (+ 2360us) service_pool.cc:225] Handling call
0112 20:51:42.649054 (+1063904us) ts_tablet_manager.cc:1145] Deleting tablet ab9be1f065ec4605a0a3fe69770793d0
0112 20:51:42.649064 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.649186 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1063671}
I20250112 20:51:42.654292 1186 ts_tablet_manager.cc:1918] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:42.654549 1185 tablet_replica.cc:331] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:42.654819 1185 raft_consensus.cc:2238] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:42.656011 710 consensus_peers.cc:487] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64e1c32cab2c4b438d3ff4a7e674feeb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:42.656456 1185 raft_consensus.cc:2267] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.657552 1185 ts_tablet_manager.cc:1905] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.662210 1163 tablet_bootstrap.cc:492] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.658229 1164 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.662392 1163 ts_tablet_manager.cc:1397] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.001s
I20250112 20:51:42.662526 1164 ts_tablet_manager.cc:1397] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.002s
I20250112 20:51:42.663203 1163 raft_consensus.cc:357] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.663396 1163 raft_consensus.cc:383] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.663450 1163 raft_consensus.cc:738] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.663245 1164 raft_consensus.cc:357] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.663658 1164 raft_consensus.cc:383] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.663749 1164 raft_consensus.cc:738] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.663648 1163 consensus_queue.cc:260] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.664212 1163 ts_tablet_manager.cc:1428] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:42.664902 1163 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.666258 1164 consensus_queue.cc:260] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.666893 1164 ts_tablet_manager.cc:1428] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:42.667865 1186 log.cc:1198] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:42.668169 1164 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.670209 1186 ts_tablet_manager.cc:1939] T d570daec845f40778c48ff3f2e2e30cb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:42.673677 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 772) took 1380 ms. Trace:
I20250112 20:51:42.674643 1164 tablet_bootstrap.cc:654] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.674777 1186 rpcz_store.cc:276] 0112 20:51:41.292697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.293214 (+ 517us) service_pool.cc:225] Handling call
0112 20:51:42.624523 (+1331309us) ts_tablet_manager.cc:1145] Deleting tablet d570daec845f40778c48ff3f2e2e30cb
0112 20:51:42.624533 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.649625 (+ 25092us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.654193 (+ 4568us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:42.673656 (+ 19463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321126}
I20250112 20:51:42.675017 883 tablet_service.cc:1514] Processing DeleteTablet for tablet fae080f8e4b7436b9166c85604a77dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.676268 1186 tablet_replica.cc:331] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:42.675846 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.677147 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet fae080f8e4b7436b9166c85604a77dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.677536 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.677701 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 36f95046c87846b2aa7e15430c1480b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.678087 1042 raft_consensus.cc:3054] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.679445 1186 raft_consensus.cc:2238] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:42.675697 881 tablet_service.cc:1514] Processing DeleteTablet for tablet f9055531da4c453fa722eb7c78fa0e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.684423 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fae080f8e4b7436b9166c85604a77dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.686422 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.686640 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 36f95046c87846b2aa7e15430c1480b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.687053 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f9055531da4c453fa722eb7c78fa0e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.688238 1178 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.688423 1178 ts_tablet_manager.cc:1397] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.004s
I20250112 20:51:42.689213 1178 raft_consensus.cc:357] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.690296 1178 raft_consensus.cc:383] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.690477 1178 raft_consensus.cc:738] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.690860 1178 consensus_queue.cc:260] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.691627 1178 ts_tablet_manager.cc:1428] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:42.692605 1178 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.676755 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 36f95046c87846b2aa7e15430c1480b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.695827 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f9055531da4c453fa722eb7c78fa0e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.698076 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c697bba271f4455956b9a2511155b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.698329 2447 tablet_replica.cc:331] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:42.698559 2447 raft_consensus.cc:2238] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.698726 2447 raft_consensus.cc:2267] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.679648 1186 pending_rounds.cc:62] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:42.698823 1186 pending_rounds.cc:69] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585058701144064 op_type: NO_OP noop_request { }
I20250112 20:51:42.698921 1186 raft_consensus.cc:2883] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:42.698997 1186 raft_consensus.cc:2267] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.699317 2447 ts_tablet_manager.cc:1905] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.700541 1178 tablet_bootstrap.cc:654] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.702325 1164 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.702461 1164 ts_tablet_manager.cc:1397] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
I20250112 20:51:42.703186 1164 raft_consensus.cc:357] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.703382 1164 raft_consensus.cc:383] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.703444 1164 raft_consensus.cc:738] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.705209 1163 tablet_bootstrap.cc:654] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.707036 2447 ts_tablet_manager.cc:1918] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:42.707191 2447 log.cc:1198] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8c697bba271f4455956b9a2511155b79
I20250112 20:51:42.707346 1185 ts_tablet_manager.cc:1918] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:42.707664 2447 ts_tablet_manager.cc:1939] T 8c697bba271f4455956b9a2511155b79 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:42.709993 450 catalog_manager.cc:5526] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.710805 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:42.711095 450 catalog_manager.cc:5526] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.711694 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:42.713126 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c697bba271f4455956b9a2511155b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.712469 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c697bba271f4455956b9a2511155b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.713371 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.716010 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0a87da1ea4054ffc81b9c1c792f0cc5d
I20250112 20:51:42.716472 452 catalog_manager.cc:6596] ReplaceTablet: tablet 0a87da1ea4054ffc81b9c1c792f0cc5d deleted and replaced by tablet ff84b2dae9fe4cd5b3bcf625a7d3e829
I20250112 20:51:42.717528 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.718032 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 2440e0bdf8ad4ab7805c892e28834acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:42 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.720506 772 raft_consensus.cc:1270] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:42.721014 2415 consensus_queue.cc:1035] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:42.721316 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4f34aa32d6474561988b2cf3d0e8fd3b (table default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]) successfully deleted
I20250112 20:51:42.722859 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 913149098822406e8d4ca5a440c9afd7 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:42.733152 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 35dcd1a120684446bbe19303fb8416ef because the tablet was not found. No further retry: Not found: Tablet not found: 35dcd1a120684446bbe19303fb8416ef
W20250112 20:51:42.733369 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 762388f687dd4b05ae6ffdaf8c64c999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 762388f687dd4b05ae6ffdaf8c64c999 already in progress: creating tablet
I20250112 20:51:42.712448 1164 consensus_queue.cc:260] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.733731 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8b7b788acf8047acb34ca2596bbf141b (table default.Table_c09ffcd01d594a1d8b065295cf69258a [id=22e212eacf0449b48069d0c9b7cda461]) successfully deleted
I20250112 20:51:42.734213 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f720bd9710d2476d9632699a2732cb05 (table default.Table_5860b3a08e52455f9708d82fe27361b9 [id=32ea0a446f8e4aa2b87dfd1c5b691419]) successfully deleted
I20250112 20:51:42.734678 1178 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
W20250112 20:51:42.734987 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1a505986bc6146829cbafd0de6682d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a505986bc6146829cbafd0de6682d93 already in progress: creating tablet
W20250112 20:51:42.735217 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 49bfc4fbe78745049699c8383c724f38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bfc4fbe78745049699c8383c724f38 already in progress: creating tablet
W20250112 20:51:42.735471 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cbdc5c117ca9437091a1667642c7c846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdc5c117ca9437091a1667642c7c846 already in progress: creating tablet
I20250112 20:51:42.735750 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 70653110aec74727acec9d63c2eb0f05 (table default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]) successfully deleted
W20250112 20:51:42.736006 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ab9be1f065ec4605a0a3fe69770793d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab9be1f065ec4605a0a3fe69770793d0 already in progress: creating tablet
I20250112 20:51:42.736064 1164 ts_tablet_manager.cc:1428] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
W20250112 20:51:42.736234 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b519b0dc51a240629b6376767fc19e5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b519b0dc51a240629b6376767fc19e5d already in progress: creating tablet
I20250112 20:51:42.736483 1178 ts_tablet_manager.cc:1397] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20250112 20:51:42.737030 1164 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.741880 1185 log.cc:1198] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:42.742439 1185 ts_tablet_manager.cc:1939] T 64e1c32cab2c4b438d3ff4a7e674feeb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:42.743220 1178 raft_consensus.cc:357] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.749908 1178 raft_consensus.cc:383] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.750171 1178 raft_consensus.cc:738] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.751575 1185 tablet_replica.cc:331] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:42.751804 1178 consensus_queue.cc:260] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.753021 1178 ts_tablet_manager.cc:1428] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:42.753495 2415 raft_consensus.cc:491] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.753823 2415 raft_consensus.cc:513] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.754369 2415 leader_election.cc:290] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.755295 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.755882 847 leader_election.cc:343] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.757310 1186 ts_tablet_manager.cc:1905] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.759336 2484 raft_consensus.cc:491] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.759469 2484 raft_consensus.cc:513] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.760198 2484 leader_election.cc:290] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.762501 1178 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.762848 1185 raft_consensus.cc:2238] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:42.763263 1185 raft_consensus.cc:2267] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:42.764225 1185 ts_tablet_manager.cc:1905] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:42.758847 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.753279 1164 tablet_bootstrap.cc:654] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.765349 774 raft_consensus.cc:2463] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.765820 847 leader_election.cc:304] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.768919 2371 raft_consensus.cc:2798] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.769027 2371 raft_consensus.cc:491] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.769119 2371 raft_consensus.cc:3054] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.772066 2371 raft_consensus.cc:513] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.773113 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:42.774132 847 leader_election.cc:343] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.774776 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.774950 774 raft_consensus.cc:3054] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:42.775154 983 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 267ebb3c30194d9fa6a092d9bec4eadf
W20250112 20:51:42.775332 983 leader_election.cc:343] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:42.775408 983 leader_election.cc:304] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:42.775627 2484 raft_consensus.cc:2743] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:42.779564 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d3ea347fd2694016a3cd3721b0cab608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3ea347fd2694016a3cd3721b0cab608 already in progress: creating tablet
W20250112 20:51:42.779866 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b37e9e95c6b745ccaba5446b04abcc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37e9e95c6b745ccaba5446b04abcc9d already in progress: creating tablet
W20250112 20:51:42.780089 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 4633e6870ccd4ce5b2458bb22155ba42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4633e6870ccd4ce5b2458bb22155ba42 already in progress: creating tablet
W20250112 20:51:42.781622 440 catalog_manager.cc:4945] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): alter failed for tablet d0829e9dd5854961b5402b5710624612 (table default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]),no further retry: Not found: Tablet not found: d0829e9dd5854961b5402b5710624612
I20250112 20:51:42.781857 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]) successfully deleted
W20250112 20:51:42.782055 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet fae080f8e4b7436b9166c85604a77dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fae080f8e4b7436b9166c85604a77dfa already in progress: creating tablet
W20250112 20:51:42.782248 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5f264d84ae4846d1afaaabc41d5789f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f264d84ae4846d1afaaabc41d5789f4 already in progress: creating tablet
W20250112 20:51:42.782423 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 36f95046c87846b2aa7e15430c1480b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f95046c87846b2aa7e15430c1480b5 already in progress: creating tablet
W20250112 20:51:42.782615 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f9055531da4c453fa722eb7c78fa0e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9055531da4c453fa722eb7c78fa0e78 already in progress: creating tablet
I20250112 20:51:42.782938 2566 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:42.783218 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 267ebb3c30194d9fa6a092d9bec4eadf (table default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]) successfully deleted
W20250112 20:51:42.783432 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 762388f687dd4b05ae6ffdaf8c64c999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 762388f687dd4b05ae6ffdaf8c64c999 already in progress: creating tablet
I20250112 20:51:42.783735 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d570daec845f40778c48ff3f2e2e30cb (table default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]) successfully deleted
I20250112 20:51:42.784238 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d with delete_type TABLET_DATA_DELETED (replaced by tablet ff84b2dae9fe4cd5b3bcf625a7d3e829) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.784238 2371 leader_election.cc:290] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.784060 2567 tablet.cc:1721] T 9cf1dc64f1964202bfa8169388a34781 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:42.791674 2571 tablet.cc:1721] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:42.795243 774 raft_consensus.cc:2463] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.795751 847 leader_election.cc:304] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.796048 2371 raft_consensus.cc:2798] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.796324 2371 raft_consensus.cc:695] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.796628 2371 consensus_queue.cc:237] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.801887 1185 ts_tablet_manager.cc:1918] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:42.802465 1185 log.cc:1198] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8c697bba271f4455956b9a2511155b79
I20250112 20:51:42.804668 1185 ts_tablet_manager.cc:1939] T 8c697bba271f4455956b9a2511155b79 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:42.804911 1163 tablet_bootstrap.cc:492] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.805059 1163 ts_tablet_manager.cc:1397] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.140s user 0.006s sys 0.000s
I20250112 20:51:42.805935 2568 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:42.808801 2569 tablet.cc:1721] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:42.809439 1163 raft_consensus.cc:357] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.809711 1163 raft_consensus.cc:383] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.810081 1163 raft_consensus.cc:738] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.810367 1163 consensus_queue.cc:260] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.810952 1163 ts_tablet_manager.cc:1428] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:42.811350 1164 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.811559 1164 ts_tablet_manager.cc:1397] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.008s
I20250112 20:51:42.812853 1164 raft_consensus.cc:357] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.813197 1164 raft_consensus.cc:383] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.813335 1164 raft_consensus.cc:738] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.813694 1164 consensus_queue.cc:260] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.815850 1164 ts_tablet_manager.cc:1428] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:42.817061 1163 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.817708 2572 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:42.819742 1164 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.829231 1164 tablet_bootstrap.cc:654] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.840413 1163 tablet_bootstrap.cc:654] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.842295 1164 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:42.842545 1164 ts_tablet_manager.cc:1397] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20250112 20:51:42.843402 2108 raft_consensus.cc:491] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.843683 2108 raft_consensus.cc:513] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.845218 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab635fec8a64efd8e0c527d913ccd7d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.845770 847 leader_election.cc:343] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.845794 2108 leader_election.cc:290] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.846426 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab635fec8a64efd8e0c527d913ccd7d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:42.846599 770 raft_consensus.cc:2463] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:42.846577 1164 raft_consensus.cc:357] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.846993 1164 raft_consensus.cc:383] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.847086 1164 raft_consensus.cc:738] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.847321 847 leader_election.cc:304] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.847590 2108 raft_consensus.cc:2798] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.847735 2108 raft_consensus.cc:491] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.847925 2108 raft_consensus.cc:3054] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.851387 1178 tablet_bootstrap.cc:654] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.854100 2108 raft_consensus.cc:513] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.856153 2108 leader_election.cc:290] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:42.856736 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab635fec8a64efd8e0c527d913ccd7d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:42.856822 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab635fec8a64efd8e0c527d913ccd7d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:42.857002 770 raft_consensus.cc:3054] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.856921 455 catalog_manager.cc:5526] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.857390 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:42.857328 1164 consensus_queue.cc:260] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.858621 1164 ts_tablet_manager.cc:1428] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250112 20:51:42.858884 1164 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:42.860100 770 raft_consensus.cc:2463] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:42.860687 847 leader_election.cc:304] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:42.860953 847 leader_election.cc:343] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.861510 2108 raft_consensus.cc:2798] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:42.861631 2108 raft_consensus.cc:695] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:42.861764 2208 raft_consensus.cc:491] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.861856 2208 raft_consensus.cc:513] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.861853 2108 consensus_queue.cc:237] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.862375 2208 leader_election.cc:290] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:42.865262 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:42.865458 913 raft_consensus.cc:2463] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:42.865482 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:42.866099 710 leader_election.cc:343] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:42.866291 710 leader_election.cc:304] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:42.870707 2208 raft_consensus.cc:2798] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:42.870795 2208 raft_consensus.cc:491] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:42.870853 2208 raft_consensus.cc:3054] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.874397 1178 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.874589 1178 ts_tablet_manager.cc:1397] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.009s
I20250112 20:51:42.875324 1178 raft_consensus.cc:357] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.875666 1178 raft_consensus.cc:383] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.875777 1178 raft_consensus.cc:738] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.876039 1178 consensus_queue.cc:260] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.876663 1178 ts_tablet_manager.cc:1428] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:42.876958 1178 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.878525 1178 tablet_bootstrap.cc:654] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.883708 1163 tablet_bootstrap.cc:492] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.883853 1163 ts_tablet_manager.cc:1397] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.002s
I20250112 20:51:42.885201 1163 raft_consensus.cc:357] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.885425 1163 raft_consensus.cc:383] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.885480 1163 raft_consensus.cc:738] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.885684 1163 consensus_queue.cc:260] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.895298 1163 ts_tablet_manager.cc:1428] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250112 20:51:42.903910 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79] alter complete (version 1)
I20250112 20:51:42.907498 1163 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.909711 1163 tablet_bootstrap.cc:654] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.913223 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_1e8f92a0b14c40829a4e275e8327e838 [id=3a4de1859ffd4db3931ce68febdd8836] alter complete (version 4)
I20250112 20:51:42.914330 1178 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:42.914528 1178 ts_tablet_manager.cc:1397] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250112 20:51:42.915385 1178 raft_consensus.cc:357] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.915671 1178 raft_consensus.cc:383] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.915771 1178 raft_consensus.cc:738] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.916021 1178 consensus_queue.cc:260] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.916303 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d with delete_type TABLET_DATA_DELETED (replaced by tablet ff84b2dae9fe4cd5b3bcf625a7d3e829) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:42.916821 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d with delete_type TABLET_DATA_DELETED (replaced by tablet ff84b2dae9fe4cd5b3bcf625a7d3e829) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:42.917958 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 61c7185087bb463aaa23eb21d696991c (DEFAULT_TABLE table=default.Table_8fde83dfa40f4a8783c855991337d74a [id=cabe0a5683d14239a22eb427caecbe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.918319 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c7185087bb463aaa23eb21d696991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.919472 747 tablet_service.cc:1467] Processing CreateTablet for tablet 61c7185087bb463aaa23eb21d696991c (DEFAULT_TABLE table=default.Table_8fde83dfa40f4a8783c855991337d74a [id=cabe0a5683d14239a22eb427caecbe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.920210 883 tablet_service.cc:1467] Processing CreateTablet for tablet 61c7185087bb463aaa23eb21d696991c (DEFAULT_TABLE table=default.Table_8fde83dfa40f4a8783c855991337d74a [id=cabe0a5683d14239a22eb427caecbe9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.920424 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a7d0ae4334e148f399c9d23df2d2e (DEFAULT_TABLE table=default.Table_a38079a1fd8c479a839b50c4c84eca80 [id=6568d3bfe15e4ff1801956e3d786684a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.920794 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5a7d0ae4334e148f399c9d23df2d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.920944 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c7185087bb463aaa23eb21d696991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.921481 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a7d0ae4334e148f399c9d23df2d2e (DEFAULT_TABLE table=default.Table_a38079a1fd8c479a839b50c4c84eca80 [id=6568d3bfe15e4ff1801956e3d786684a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.921811 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5a7d0ae4334e148f399c9d23df2d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.922204 1042 raft_consensus.cc:3054] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:42.922456 1178 ts_tablet_manager.cc:1428] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:42.922730 1178 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:42.924887 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.925457 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.925710 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.925923 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.926373 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:42.927018 881 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a7d0ae4334e148f399c9d23df2d2e (DEFAULT_TABLE table=default.Table_a38079a1fd8c479a839b50c4c84eca80 [id=6568d3bfe15e4ff1801956e3d786684a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:42.927353 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5a7d0ae4334e148f399c9d23df2d2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:42.927585 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet fae080f8e4b7436b9166c85604a77dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fae080f8e4b7436b9166c85604a77dfa already in progress: creating tablet
W20250112 20:51:42.927865 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5f264d84ae4846d1afaaabc41d5789f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f264d84ae4846d1afaaabc41d5789f4 already in progress: creating tablet
W20250112 20:51:42.928079 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 36f95046c87846b2aa7e15430c1480b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f95046c87846b2aa7e15430c1480b5 already in progress: creating tablet
W20250112 20:51:42.928267 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f9055531da4c453fa722eb7c78fa0e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9055531da4c453fa722eb7c78fa0e78 already in progress: creating tablet
I20250112 20:51:42.928465 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]) successfully deleted
W20250112 20:51:42.928642 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2440e0bdf8ad4ab7805c892e28834acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2440e0bdf8ad4ab7805c892e28834acd already in progress: creating tablet
W20250112 20:51:42.928953 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a87da1ea4054ffc81b9c1c792f0cc5d already in progress: creating tablet
I20250112 20:51:42.932999 1178 tablet_bootstrap.cc:654] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.946727 1163 tablet_bootstrap.cc:492] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:42.946969 1163 ts_tablet_manager.cc:1397] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20250112 20:51:42.947948 1163 raft_consensus.cc:357] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.948246 1163 raft_consensus.cc:383] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:42.948756 1163 raft_consensus.cc:738] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:42.949087 1163 consensus_queue.cc:260] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:42.949630 1163 ts_tablet_manager.cc:1428] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 20:51:42.949939 1163 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:42.951726 1163 tablet_bootstrap.cc:654] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:42.958931 2326 raft_consensus.cc:491] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.959116 2326 raft_consensus.cc:513] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:42.964681 847 consensus_peers.cc:487] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:42.965629 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu pid 1218
I20250112 20:51:42.968691 2563 raft_consensus.cc:491] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:42.968844 2563 raft_consensus.cc:513] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:42.969463 2563 leader_election.cc:290] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:42.970471 983 leader_election.cc:343] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 35dcd1a120684446bbe19303fb8416ef
W20250112 20:51:42.973560 847 consensus_peers.cc:487] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c697bba271f4455956b9a2511155b79. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:42.980757 2059 raft_consensus.cc:491] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.014356 1186 ts_tablet_manager.cc:1918] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.014489 1186 log.cc:1198] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8b7b788acf8047acb34ca2596bbf141b
I20250112 20:51:43.014882 1186 ts_tablet_manager.cc:1939] T 8b7b788acf8047acb34ca2596bbf141b P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.015298 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 773) took 1722 ms. Trace:
I20250112 20:51:43.015468 1186 rpcz_store.cc:276] 0112 20:51:41.292770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.292971 (+ 201us) service_pool.cc:225] Handling call
0112 20:51:42.676204 (+1383233us) ts_tablet_manager.cc:1145] Deleting tablet 8b7b788acf8047acb34ca2596bbf141b
0112 20:51:42.676229 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:42.852758 (+176529us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.014237 (+161479us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.015282 (+ 1045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361219}
I20250112 20:51:43.015935 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 784) took 1615 ms. Trace:
I20250112 20:51:43.016031 1186 rpcz_store.cc:276] 0112 20:51:41.400095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.400220 (+ 125us) service_pool.cc:225] Handling call
0112 20:51:43.015776 (+1615556us) ts_tablet_manager.cc:1145] Deleting tablet 1a505986bc6146829cbafd0de6682d93
0112 20:51:43.015786 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.015915 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615370}
W20250112 20:51:43.021291 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8098069639a2435ab500908db9ecd12a (table default.Table_3886e5ca0a0b4c71ac1ade9fcf9d8ab2 [id=554666ac0524458dbb36e43d52dfbdaa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8098069639a2435ab500908db9ecd12a consensus state has no leader
I20250112 20:51:43.024207 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c7185087bb463aaa23eb21d696991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:42.980973 2059 raft_consensus.cc:513] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.027839 2326 leader_election.cc:290] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.028493 2059 leader_election.cc:290] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.034688 1164 tablet_bootstrap.cc:654] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.056918 2108 raft_consensus.cc:491] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.057075 2108 raft_consensus.cc:513] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.057811 2108 leader_election.cc:290] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.058717 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:43.059371 847 leader_election.cc:343] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.060434 2208 raft_consensus.cc:513] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.061537 2208 leader_election.cc:290] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.078928 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.079161 775 raft_consensus.cc:2463] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:43.079722 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.080163 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235faeb3c35d46d2882ffcd8cb63ddb8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:43.080797 983 leader_election.cc:343] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:43.080651 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_65cffe4814de46449661f915c2371007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:43.080893 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.080924 983 leader_election.cc:304] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.081405 2563 raft_consensus.cc:2743] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:43.081490 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65cffe4814de46449661f915c2371007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.084707 847 leader_election.cc:304] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.085208 2108 raft_consensus.cc:2798] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.085282 2108 raft_consensus.cc:491] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.085335 2108 raft_consensus.cc:3054] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.087890 2563 raft_consensus.cc:491] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.087999 2563 raft_consensus.cc:513] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.088572 2563 leader_election.cc:290] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.089354 2108 raft_consensus.cc:513] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.090149 2108 leader_election.cc:290] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.091143 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:43.091588 847 leader_election.cc:343] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.091758 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ce095329e25c4874a1e151a4f5d84c72" } new_table_name: "default.Table_7ffaf1ff93694b688350c7dfb482f7fe" modify_external_catalogs: true
I20250112 20:51:43.092526 1164 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.092674 1164 ts_tablet_manager.cc:1397] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.234s user 0.003s sys 0.007s
I20250112 20:51:43.093667 1164 raft_consensus.cc:357] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.093832 1164 raft_consensus.cc:383] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.093880 1164 raft_consensus.cc:738] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.094107 1164 consensus_queue.cc:260] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.094523 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_acf981f928ef4a12a31795dec2c5256e" } new_table_name: "default.Table_5a3edf19ce64498bbf0f8518ad5c4f68" modify_external_catalogs: true
I20250112 20:51:43.096972 1164 ts_tablet_manager.cc:1428] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:43.097258 1164 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.098848 1164 tablet_bootstrap.cc:654] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.099962 710 leader_election.cc:343] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.100140 710 leader_election.cc:343] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.100250 710 leader_election.cc:343] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.100399 710 consensus_peers.cc:487] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:43.101044 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:43.101226 772 raft_consensus.cc:3054] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:43.104930 983 leader_election.cc:343] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 3bbb679585fc44699553fe85ea044566
I20250112 20:51:43.104961 772 raft_consensus.cc:2463] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:43.105401 847 leader_election.cc:304] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.105835 2108 raft_consensus.cc:2798] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.106070 2108 raft_consensus.cc:695] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:43.106329 2108 consensus_queue.cc:237] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.112275 2563 raft_consensus.cc:491] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.112407 2563 raft_consensus.cc:513] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.113130 2563 leader_election.cc:290] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.120843 2563 raft_consensus.cc:491] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.120971 2563 raft_consensus.cc:513] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:43.121526 2563 leader_election.cc:290] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:43.124064 983 leader_election.cc:343] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:43.127667 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.127871 911 raft_consensus.cc:3054] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.127898 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235faeb3c35d46d2882ffcd8cb63ddb8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.128059 910 raft_consensus.cc:2463] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.129257 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 785) took 1729 ms. Trace:
I20250112 20:51:43.129343 1186 rpcz_store.cc:276] 0112 20:51:41.400199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.400430 (+ 231us) service_pool.cc:225] Handling call
0112 20:51:43.016238 (+1615808us) ts_tablet_manager.cc:1145] Deleting tablet cbdc5c117ca9437091a1667642c7c846
0112 20:51:43.127396 (+111158us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.129242 (+ 1846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615703}
I20250112 20:51:43.130564 911 raft_consensus.cc:2463] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.127866 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903027fe49e34a308108356ba89a9c3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.131543 1178 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.131652 1178 ts_tablet_manager.cc:1397] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.209s user 0.043s sys 0.071s
I20250112 20:51:43.132354 1178 raft_consensus.cc:357] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.132504 1178 raft_consensus.cc:383] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.132548 1178 raft_consensus.cc:738] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.132709 1178 consensus_queue.cc:260] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.137466 1178 ts_tablet_manager.cc:1428] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:43.137738 1178 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.141644 1178 tablet_bootstrap.cc:654] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.144572 983 leader_election.cc:343] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:43.144692 983 leader_election.cc:304] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:43.145071 983 leader_election.cc:343] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 3bbb679585fc44699553fe85ea044566
I20250112 20:51:43.145150 983 leader_election.cc:304] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:43.145421 983 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:43.145555 2564 raft_consensus.cc:2743] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.145673 2564 raft_consensus.cc:2743] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.151957 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.152573 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.153601 1178 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.153728 1178 ts_tablet_manager.cc:1397] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250112 20:51:43.153733 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e132d9ac027d44bc8aa20fbdacd57a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:43.154435 1178 raft_consensus.cc:357] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.154640 1178 raft_consensus.cc:383] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.154692 1178 raft_consensus.cc:738] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.154850 1178 consensus_queue.cc:260] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.155382 1178 ts_tablet_manager.cc:1428] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:43.155608 1178 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.157151 1178 tablet_bootstrap.cc:654] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.159075 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e132d9ac027d44bc8aa20fbdacd57a62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:43.160730 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1aa4249c43bb41fe8dec95e0c910a2c5 consensus state has no leader
I20250112 20:51:43.161579 1178 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.161785 1178 ts_tablet_manager.cc:1397] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250112 20:51:43.163493 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 786) took 1756 ms. Trace:
I20250112 20:51:43.163618 1186 rpcz_store.cc:276] 0112 20:51:41.406947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.407005 (+ 58us) service_pool.cc:225] Handling call
0112 20:51:43.129568 (+1722563us) ts_tablet_manager.cc:1145] Deleting tablet 49bfc4fbe78745049699c8383c724f38
0112 20:51:43.163374 (+ 33806us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.163475 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722411}
I20250112 20:51:43.164078 1186 tablet_replica.cc:331] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.164481 1186 raft_consensus.cc:2238] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.164709 1186 raft_consensus.cc:2267] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.165431 1186 ts_tablet_manager.cc:1905] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.175343 2521 raft_consensus.cc:491] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.175474 2521 raft_consensus.cc:513] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.176010 2521 leader_election.cc:290] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.176718 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4561956c97824824a55283a8da21929e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:43.177412 847 leader_election.cc:343] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.181358 1163 tablet_bootstrap.cc:492] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.181546 1163 ts_tablet_manager.cc:1397] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.232s user 0.013s sys 0.027s
I20250112 20:51:43.182264 1163 raft_consensus.cc:357] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.182513 1163 raft_consensus.cc:383] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.182600 1163 raft_consensus.cc:738] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.182869 1163 consensus_queue.cc:260] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:43.185190 983 leader_election.cc:343] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:43.185305 983 leader_election.cc:304] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.185721 2563 raft_consensus.cc:2743] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.187450 710 leader_election.cc:304] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.187805 2585 raft_consensus.cc:2798] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.187927 2585 raft_consensus.cc:491] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.188086 2585 raft_consensus.cc:3054] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.188719 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4561956c97824824a55283a8da21929e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.188953 775 raft_consensus.cc:2463] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:43.192494 2585 raft_consensus.cc:513] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.193156 2585 leader_election.cc:290] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:43.193434 2588 log.cc:926] Time spent T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250112 20:51:43.194283 710 leader_election.cc:304] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.194808 710 leader_election.cc:304] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:43.196480 847 consensus_peers.cc:487] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:43.195035 2585 raft_consensus.cc:2798] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.197113 2208 raft_consensus.cc:2743] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.197971 2585 raft_consensus.cc:695] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.198348 2585 consensus_queue.cc:237] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.198964 847 leader_election.cc:304] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.199486 2108 raft_consensus.cc:2798] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.199661 2108 raft_consensus.cc:491] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.199784 2108 raft_consensus.cc:3054] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.203485 2108 raft_consensus.cc:513] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.203996 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8b7b788acf8047acb34ca2596bbf141b (table default.Table_c09ffcd01d594a1d8b065295cf69258a [id=22e212eacf0449b48069d0c9b7cda461]) successfully deleted
W20250112 20:51:43.204246 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1a505986bc6146829cbafd0de6682d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a505986bc6146829cbafd0de6682d93 already in progress: creating tablet
W20250112 20:51:43.204458 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cbdc5c117ca9437091a1667642c7c846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdc5c117ca9437091a1667642c7c846 already in progress: creating tablet
I20250112 20:51:43.204524 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4561956c97824824a55283a8da21929e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:43.204679 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 49bfc4fbe78745049699c8383c724f38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bfc4fbe78745049699c8383c724f38 already in progress: creating tablet
I20250112 20:51:43.149886 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_30fc56d0c044444a9a07861b6650fac6" } new_table_name: "default.Table_b5e7418fef2d44709ff22f9961666ee3" modify_external_catalogs: true
W20250112 20:51:43.205404 847 leader_election.cc:343] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.205849 2108 leader_election.cc:290] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:43.206465 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34608d460c1347a9895849ea77776299 has no consensus state
I20250112 20:51:43.206861 1163 ts_tablet_manager.cc:1428] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20250112 20:51:43.207839 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_968d00519a1942b78b337dc9f4980744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:43.207993 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f7662b40ae1447808abeaa1501705e2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.208521 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7662b40ae1447808abeaa1501705e2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:43.208559 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_968d00519a1942b78b337dc9f4980744 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.210469 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fe56f26bf2934585834bfaf76f5a1d08" } modify_external_catalogs: true
I20250112 20:51:43.210659 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fe56f26bf2934585834bfaf76f5a1d08" } modify_external_catalogs: true
I20250112 20:51:43.199858 1164 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.212419 1164 ts_tablet_manager.cc:1397] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.000s
I20250112 20:51:43.213042 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ee975b071729460b863016012a403cb4" } modify_external_catalogs: true
I20250112 20:51:43.213248 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ee975b071729460b863016012a403cb4" } modify_external_catalogs: true
I20250112 20:51:43.213380 1164 raft_consensus.cc:357] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.213624 1164 raft_consensus.cc:383] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.213683 1164 raft_consensus.cc:738] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.213081 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235faeb3c35d46d2882ffcd8cb63ddb8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.213917 1164 consensus_queue.cc:260] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.214475 1164 ts_tablet_manager.cc:1428] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:43.214752 1164 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.215080 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235faeb3c35d46d2882ffcd8cb63ddb8" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.215243 907 raft_consensus.cc:3054] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.217810 907 raft_consensus.cc:2463] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.218183 1164 tablet_bootstrap.cc:654] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.221266 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]) successfully deleted
W20250112 20:51:43.221545 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2440e0bdf8ad4ab7805c892e28834acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2440e0bdf8ad4ab7805c892e28834acd already in progress: creating tablet
W20250112 20:51:43.221752 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a87da1ea4054ffc81b9c1c792f0cc5d already in progress: creating tablet
I20250112 20:51:43.233392 1163 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.240319 710 leader_election.cc:304] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.241195 2208 raft_consensus.cc:2798] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.241331 2208 raft_consensus.cc:695] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.241552 2208 consensus_queue.cc:237] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.243415 2208 consensus_queue.cc:1035] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:43.244073 710 leader_election.cc:343] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.246665 1178 raft_consensus.cc:357] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.247457 1163 tablet_bootstrap.cc:654] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.249610 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4561956c97824824a55283a8da21929e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:43.249778 772 raft_consensus.cc:3054] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.252528 772 raft_consensus.cc:2463] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:43.246876 1178 raft_consensus.cc:383] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.254619 1178 raft_consensus.cc:738] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.255143 1178 consensus_queue.cc:260] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.256327 1178 ts_tablet_manager.cc:1428] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 20:51:43.257499 2563 raft_consensus.cc:491] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.257753 1178 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.257664 2563 raft_consensus.cc:513] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.258285 2563 leader_election.cc:290] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.258898 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.259207 1178 tablet_bootstrap.cc:654] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.263747 2108 consensus_queue.cc:1035] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.264938 847 leader_election.cc:304] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.265017 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30156185f6104eada87b936fc1829df9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.265446 2521 raft_consensus.cc:2798] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.265554 2521 raft_consensus.cc:695] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:43.265759 2521 consensus_queue.cc:237] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.275358 1164 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.279572 1164 ts_tablet_manager.cc:1397] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.003s
I20250112 20:51:43.280530 1164 raft_consensus.cc:357] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.280825 1164 raft_consensus.cc:383] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.280926 1164 raft_consensus.cc:738] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.281167 1164 consensus_queue.cc:260] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.281773 1164 ts_tablet_manager.cc:1428] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:43.282074 1164 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.284534 1164 tablet_bootstrap.cc:654] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.288399 2208 raft_consensus.cc:491] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.288595 2208 raft_consensus.cc:513] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.289204 2208 leader_election.cc:290] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.292510 983 leader_election.cc:304] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.292984 2563 raft_consensus.cc:2743] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.295854 1163 tablet_bootstrap.cc:492] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.296037 1163 ts_tablet_manager.cc:1397] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250112 20:51:43.296766 1163 raft_consensus.cc:357] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.297058 1163 raft_consensus.cc:383] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.297153 1163 raft_consensus.cc:738] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.299919 1163 consensus_queue.cc:260] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.300381 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d706f88f14ab4fd9a459944863afaacd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.301381 1163 ts_tablet_manager.cc:1428] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 20:51:43.300830 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d706f88f14ab4fd9a459944863afaacd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.302057 908 raft_consensus.cc:2463] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.295109 2521 consensus_queue.cc:1035] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250112 20:51:43.278507 1186 ts_tablet_manager.cc:1918] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:43.305696 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d80791d8582446b89adf212d3d6507eb consensus state has no leader
I20250112 20:51:43.330617 1186 log.cc:1198] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:43.337575 1164 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.337944 1164 ts_tablet_manager.cc:1397] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20250112 20:51:43.338462 1186 ts_tablet_manager.cc:1939] T 4f34aa32d6474561988b2cf3d0e8fd3b P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.341064 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 787) took 1764 ms. Trace:
W20250112 20:51:43.339577 710 leader_election.cc:343] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.343956 1163 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.345438 1163 tablet_bootstrap.cc:654] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.346535 2571 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:43.351581 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ba86d0ef63d94a55adf79bb136ea89e8
I20250112 20:51:43.353016 744 tablet_service.cc:1514] Processing DeleteTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.352919 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2be21f44001347a1ac95041d68da6a79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.353526 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be21f44001347a1ac95041d68da6a79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.354323 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.354565 2447 tablet_replica.cc:331] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:43.354759 2447 raft_consensus.cc:2238] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.354796 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4246de22bb514400b9eb57117e36a6f9" } new_table_name: "default.Table_059de69672e845ae99db3f0615692b3f" modify_external_catalogs: true
I20250112 20:51:43.354945 2447 raft_consensus.cc:2267] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.355487 2447 ts_tablet_manager.cc:1905] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.355978 2563 raft_consensus.cc:491] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.356115 2563 raft_consensus.cc:513] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.356240 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.356683 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.357930 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_343ab21a9b504bef9d7ceb9a38d3870c" } new_table_name: "default.Table_0cf0e2c39db54427949f7adfbec95e0d" modify_external_catalogs: true
I20250112 20:51:43.352919 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_14e5a2e03fd74b87bf2afbb8a1688f30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.359203 983 leader_election.cc:343] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:43.356734 2563 leader_election.cc:290] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:43.359457 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14e5a2e03fd74b87bf2afbb8a1688f30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.361039 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_48039b63c7ee45558ce7a911e805f60a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.361517 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48039b63c7ee45558ce7a911e805f60a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.364883 710 leader_election.cc:304] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.365211 2585 raft_consensus.cc:2798] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.365290 2585 raft_consensus.cc:491] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.365351 2585 raft_consensus.cc:3054] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.365374 1185 tablet_replica.cc:331] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:43.365566 1185 raft_consensus.cc:2238] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:43.365856 1185 raft_consensus.cc:2267] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.366413 1185 ts_tablet_manager.cc:1905] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.368177 2585 raft_consensus.cc:513] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.368767 2585 leader_election.cc:290] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.368994 1186 rpcz_store.cc:276] 0112 20:51:41.576603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.576658 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:43.164028 (+1587370us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:43.164037 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.168647 (+ 4610us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.278399 (+109752us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.341032 (+ 62633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587003}
I20250112 20:51:43.369442 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4f34aa32d6474561988b2cf3d0e8fd3b (table default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]) successfully deleted
I20250112 20:51:43.369840 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 788) took 1788 ms. Trace:
I20250112 20:51:43.369915 1186 rpcz_store.cc:276] 0112 20:51:41.581581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.581632 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:43.369694 (+1788062us) ts_tablet_manager.cc:1145] Deleting tablet ab9be1f065ec4605a0a3fe69770793d0
0112 20:51:43.369706 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.369821 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787837}
I20250112 20:51:43.370189 2447 ts_tablet_manager.cc:1918] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:43.370294 2447 log.cc:1198] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ba86d0ef63d94a55adf79bb136ea89e8
W20250112 20:51:43.370321 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ab9be1f065ec4605a0a3fe69770793d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab9be1f065ec4605a0a3fe69770793d0 already in progress: creating tablet
I20250112 20:51:43.370640 2447 ts_tablet_manager.cc:1939] T ba86d0ef63d94a55adf79bb136ea89e8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:43.370662 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 796) took 1492 ms. Trace:
I20250112 20:51:43.370738 1186 rpcz_store.cc:276] 0112 20:51:41.878003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.933294 (+ 55291us) service_pool.cc:225] Handling call
0112 20:51:43.370564 (+1437270us) ts_tablet_manager.cc:1145] Deleting tablet b519b0dc51a240629b6376767fc19e5d
0112 20:51:43.370573 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.370646 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437087}
W20250112 20:51:43.371131 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b519b0dc51a240629b6376767fc19e5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b519b0dc51a240629b6376767fc19e5d already in progress: creating tablet
I20250112 20:51:43.371667 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]) successfully deleted
I20250112 20:51:43.371937 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 807) took 1430 ms. Trace:
I20250112 20:51:43.353253 881 tablet_service.cc:1514] Processing DeleteTablet for tablet ba86d0ef63d94a55adf79bb136ea89e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.327293 1178 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.372259 1178 ts_tablet_manager.cc:1397] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.002s
I20250112 20:51:43.373418 1178 raft_consensus.cc:357] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.373659 1178 raft_consensus.cc:383] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.373792 1178 raft_consensus.cc:738] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.374078 1178 consensus_queue.cc:260] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.376487 1185 ts_tablet_manager.cc:1918] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:43.376585 1185 log.cc:1198] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ba86d0ef63d94a55adf79bb136ea89e8
I20250112 20:51:43.376973 1185 ts_tablet_manager.cc:1939] T ba86d0ef63d94a55adf79bb136ea89e8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:43.378600 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]) successfully deleted
I20250112 20:51:43.381047 1042 raft_consensus.cc:3054] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.381222 1164 raft_consensus.cc:357] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.387006 2585 raft_consensus.cc:491] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.387341 2585 raft_consensus.cc:513] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.388322 2585 leader_election.cc:290] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.390218 1178 ts_tablet_manager.cc:1428] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250112 20:51:43.390636 1163 tablet_bootstrap.cc:492] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.390760 1163 ts_tablet_manager.cc:1397] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20250112 20:51:43.391495 1163 raft_consensus.cc:357] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.391675 1163 raft_consensus.cc:383] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.391731 1163 raft_consensus.cc:738] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.393473 1178 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.405386 1163 consensus_queue.cc:260] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.405967 1163 ts_tablet_manager.cc:1428] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250112 20:51:43.406250 1163 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.407676 1163 tablet_bootstrap.cc:654] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.414064 1178 tablet_bootstrap.cc:654] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.426997 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d706f88f14ab4fd9a459944863afaacd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.429659 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.430546 1164 raft_consensus.cc:383] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.430639 1164 raft_consensus.cc:738] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.432096 1164 consensus_queue.cc:260] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.437433 1164 ts_tablet_manager.cc:1428] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.099s user 0.016s sys 0.002s
I20250112 20:51:43.437849 1164 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.439257 1164 tablet_bootstrap.cc:654] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.441066 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb09123a64e744c89cb8343fc27d8d60 consensus state has no leader
W20250112 20:51:43.424245 847 consensus_peers.cc:487] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:43.449126 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 consensus state has no leader
W20250112 20:51:43.451074 847 consensus_peers.cc:487] T 64f1119a719b4c0fa47650dc1fb1c27d P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:43.453023 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e236e2ca4b654f28a0175e3f7041d249" } modify_external_catalogs: true
I20250112 20:51:43.453238 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e236e2ca4b654f28a0175e3f7041d249" } modify_external_catalogs: true
I20250112 20:51:43.429001 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d706f88f14ab4fd9a459944863afaacd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.430675 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.453841 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_72ecf6832e6c42458c48a0ffe1a66ad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:43.454041 912 raft_consensus.cc:2463] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.454387 907 raft_consensus.cc:3054] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:43.454468 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72ecf6832e6c42458c48a0ffe1a66ad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.455762 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7265385e3f78422c91c8622a9305378b" } modify_external_catalogs: true
I20250112 20:51:43.455909 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7265385e3f78422c91c8622a9305378b" } modify_external_catalogs: true
I20250112 20:51:43.431236 1186 rpcz_store.cc:276] 0112 20:51:41.941422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.941478 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:43.371339 (+1429861us) ts_tablet_manager.cc:1145] Deleting tablet b37e9e95c6b745ccaba5446b04abcc9d
0112 20:51:43.371348 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.371922 (+ 574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429629}
I20250112 20:51:43.457584 2521 raft_consensus.cc:491] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.457710 2521 raft_consensus.cc:513] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.458283 2521 leader_election.cc:290] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.458791 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 808) took 1514 ms. Trace:
I20250112 20:51:43.458865 1186 rpcz_store.cc:276] 0112 20:51:41.944771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.944846 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:43.458669 (+1513823us) ts_tablet_manager.cc:1145] Deleting tablet 4633e6870ccd4ce5b2458bb22155ba42
0112 20:51:43.458680 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.458772 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513531}
I20250112 20:51:43.459143 1186 tablet_replica.cc:331] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.459311 1186 raft_consensus.cc:2238] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.459455 1186 raft_consensus.cc:2267] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.459498 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cba467c888b84da6964f049247721310" } modify_external_catalogs: true
I20250112 20:51:43.459686 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cba467c888b84da6964f049247721310" } modify_external_catalogs: true
I20250112 20:51:43.460012 1186 ts_tablet_manager.cc:1905] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:43.462471 983 leader_election.cc:343] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:43.462604 983 leader_election.cc:304] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.462991 2563 raft_consensus.cc:2743] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.463017 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.463225 2521 raft_consensus.cc:491] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.463356 2521 raft_consensus.cc:513] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:43.463614 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b37e9e95c6b745ccaba5446b04abcc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b37e9e95c6b745ccaba5446b04abcc9d already in progress: creating tablet
W20250112 20:51:43.463894 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4633e6870ccd4ce5b2458bb22155ba42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4633e6870ccd4ce5b2458bb22155ba42 already in progress: creating tablet
I20250112 20:51:43.463959 2521 leader_election.cc:290] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:43.464380 847 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.464771 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9b7d9218094a63ad5bad03f0d27eb5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:43.465332 847 leader_election.cc:343] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.467312 2521 consensus_queue.cc:1035] T 2111e988ea634e999a1bbbf7b270153b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.469152 2521 consensus_queue.cc:1035] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.470395 2616 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:43.471819 710 leader_election.cc:343] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.472018 710 leader_election.cc:343] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.489218 907 raft_consensus.cc:2463] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.491025 2613 tablet.cc:1721] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:43.494807 710 leader_election.cc:304] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.495337 2326 raft_consensus.cc:2798] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.495477 2326 raft_consensus.cc:491] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.495597 2326 raft_consensus.cc:3054] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.497906 1164 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.498054 1164 ts_tablet_manager.cc:1397] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.002s
I20250112 20:51:43.498549 2326 raft_consensus.cc:513] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.498762 1164 raft_consensus.cc:357] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.498910 1164 raft_consensus.cc:383] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.498965 1164 raft_consensus.cc:738] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.499526 2326 leader_election.cc:290] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.501060 2585 consensus_queue.cc:1035] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.503330 452 catalog_manager.cc:5526] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:43.503786 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:43.504138 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:43.504199 1164 consensus_queue.cc:260] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.504452 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 36935e6d8c5643388ad163458e2b387c (table default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51]). Expected version 1 got 0
I20250112 20:51:43.504693 1164 ts_tablet_manager.cc:1428] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:43.505689 1164 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.508051 710 leader_election.cc:304] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.508842 2585 raft_consensus.cc:2798] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.509074 2585 raft_consensus.cc:695] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.509371 2585 consensus_queue.cc:237] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.512528 2563 raft_consensus.cc:491] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.512679 2563 raft_consensus.cc:513] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.513382 2563 leader_election.cc:290] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:43.514519 983 leader_election.cc:343] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:43.516233 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:43.517204 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_63b6aefc10aa40f7b6b171b2687a1bc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.517881 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.518783 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 34608d460c1347a9895849ea77776299 (DEFAULT_TABLE table=default.Table_7ffaf1ff93694b688350c7dfb482f7fe [id=1bbb740eb24c404181f33c5095e845c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.519193 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34608d460c1347a9895849ea77776299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.519505 1178 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.519652 1178 ts_tablet_manager.cc:1397] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.000s
I20250112 20:51:43.521275 1178 raft_consensus.cc:357] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.521611 1178 raft_consensus.cc:383] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.521684 1178 raft_consensus.cc:738] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.521919 1178 consensus_queue.cc:260] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.527530 1164 tablet_bootstrap.cc:654] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.529170 2563 raft_consensus.cc:491] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.529335 2563 raft_consensus.cc:513] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.529924 2563 leader_election.cc:290] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.530932 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.531116 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9b7d9218094a63ad5bad03f0d27eb5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.531275 772 raft_consensus.cc:2463] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:43.532079 1178 ts_tablet_manager.cc:1428] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:43.536603 1178 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.543236 1163 tablet_bootstrap.cc:492] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.543388 1163 ts_tablet_manager.cc:1397] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.137s user 0.029s sys 0.029s
W20250112 20:51:43.543985 983 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:43.544221 1163 raft_consensus.cc:357] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.544469 1163 raft_consensus.cc:383] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.544529 1163 raft_consensus.cc:738] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.544831 1163 consensus_queue.cc:260] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.547888 744 tablet_service.cc:1467] Processing CreateTablet for tablet 34608d460c1347a9895849ea77776299 (DEFAULT_TABLE table=default.Table_7ffaf1ff93694b688350c7dfb482f7fe [id=1bbb740eb24c404181f33c5095e845c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.548322 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34608d460c1347a9895849ea77776299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.550174 747 tablet_service.cc:1467] Processing CreateTablet for tablet 31c7729ffb494acb97bb2dfe356e2a0f (DEFAULT_TABLE table=default.Table_ca52e028a3b84bfe8dcb38464702a6ca [id=3f284788897545969907737ec6333966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.550767 749 tablet_service.cc:1467] Processing CreateTablet for tablet fcd5b6e74e8040239d75ec58a9dea13f (DEFAULT_TABLE table=default.Table_1c840ad1fe9c49aba52e7a3dc7281fe4 [id=bcd83f917a9d4b5c83cd1fda20e9efb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.551143 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 9850969d6a4a4610b5bdf5a780484094 (DEFAULT_TABLE table=default.Table_eaed25edb5a043188e95b48cdd144890 [id=84c2b483c928476c9049c20c31488daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.551486 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9850969d6a4a4610b5bdf5a780484094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.551427 1014 tablet_service.cc:1467] Processing CreateTablet for tablet fcd5b6e74e8040239d75ec58a9dea13f (DEFAULT_TABLE table=default.Table_1c840ad1fe9c49aba52e7a3dc7281fe4 [id=bcd83f917a9d4b5c83cd1fda20e9efb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.551757 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd5b6e74e8040239d75ec58a9dea13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.553848 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 31c7729ffb494acb97bb2dfe356e2a0f (DEFAULT_TABLE table=default.Table_ca52e028a3b84bfe8dcb38464702a6ca [id=3f284788897545969907737ec6333966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.554123 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7729ffb494acb97bb2dfe356e2a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.557650 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd5b6e74e8040239d75ec58a9dea13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.560417 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7729ffb494acb97bb2dfe356e2a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.566025 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.566196 911 raft_consensus.cc:3054] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.569578 2059 raft_consensus.cc:491] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.569706 2059 raft_consensus.cc:513] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.570245 2059 leader_election.cc:290] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.570727 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9850969d6a4a4610b5bdf5a780484094 (DEFAULT_TABLE table=default.Table_eaed25edb5a043188e95b48cdd144890 [id=84c2b483c928476c9049c20c31488daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.571141 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9850969d6a4a4610b5bdf5a780484094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.570750 753 tablet_service.cc:1467] Processing CreateTablet for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f (DEFAULT_TABLE table=default.Table_fc90d10b52794ebea4e3e19db3cc8dc8 [id=c8d1c0a4595949cf8af1f82f99f1f3c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:43.573118 983 leader_election.cc:343] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:43.573271 983 leader_election.cc:304] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.573320 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.573675 2563 raft_consensus.cc:2743] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:43.574041 983 leader_election.cc:343] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:43.574160 983 leader_election.cc:304] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.574532 2563 raft_consensus.cc:2743] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:43.575628 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 54) took 792 ms (client timeout 1000 ms). Trace:
I20250112 20:51:43.576012 1178 tablet_bootstrap.cc:654] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.575731 452 rpcz_store.cc:269] 0112 20:51:42.782866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.782934 (+ 68us) service_pool.cc:225] Handling call
0112 20:51:43.575569 (+792635us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:43.507141 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:43.507290 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:43.507300 (+ 10us) write_op.cc:435] Acquired schema lock
0112 20:51:43.507304 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:43.507412 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:43.507424 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:43.507429 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:43.507495 (+ 66us) tablet.cc:698] Row locks acquired
0112 20:51:43.507498 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:43.507537 (+ 39us) write_op.cc:270] Start()
0112 20:51:43.507573 (+ 36us) write_op.cc:276] Timestamp: P: 1736715103507480 usec, L: 0
0112 20:51:43.507577 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:43.507761 (+ 184us) log.cc:844] Serialized 527 byte log entry
0112 20:51:43.571248 (+ 63487us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.571330 (+ 82us) write_op.cc:301] APPLY: starting
0112 20:51:43.571343 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:43.571376 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:43.571398 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:43.571442 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:43.571476 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:43.571488 (+ 12us) write_op.cc:312] APPLY: finished
0112 20:51:43.571557 (+ 69us) log.cc:844] Serialized 23 byte log entry
0112 20:51:43.571615 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.571630 (+ 15us) write_op.cc:454] Released schema lock
0112 20:51:43.571637 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":35,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":365,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":714,"replication_time_us":63643}]]}
I20250112 20:51:43.577142 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 9cf1dc64f1964202bfa8169388a34781 (table default.Table_745bd9e4da3f419aa5aef5b16e9e9d58 [id=7e2c24519e17479fb708264c8543222b]). Expected version 1 got 0
I20250112 20:51:43.577422 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:43.578928 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9856e6b625d04f17b0869cee08f69142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.579156 913 raft_consensus.cc:2463] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.583097 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f (DEFAULT_TABLE table=default.Table_fc90d10b52794ebea4e3e19db3cc8dc8 [id=c8d1c0a4595949cf8af1f82f99f1f3c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.583446 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.583530 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.584049 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53518b453d224ca186fcc6b2ffab5f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.584028 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:43.584930 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9856e6b625d04f17b0869cee08f69142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.586815 1186 ts_tablet_manager.cc:1918] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:43.587002 1186 log.cc:1198] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/70653110aec74727acec9d63c2eb0f05
I20250112 20:51:43.586835 446 catalog_manager.cc:5526] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:43.587334 2059 raft_consensus.cc:491] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.587479 2059 raft_consensus.cc:513] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.587707 737 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.587916 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.588096 745 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.587575 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.588220 2059 leader_election.cc:290] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.588608 1185 tablet_replica.cc:331] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:43.588615 1186 ts_tablet_manager.cc:1939] T 70653110aec74727acec9d63c2eb0f05 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.588811 1185 raft_consensus.cc:2238] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:43.589053 1185 pending_rounds.cc:62] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:43.589099 1185 pending_rounds.cc:69] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585060025438208 op_type: NO_OP noop_request { }
I20250112 20:51:43.589215 1185 raft_consensus.cc:2883] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:43.589282 1185 raft_consensus.cc:2267] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:43.589510 847 consensus_peers.cc:487] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c697bba271f4455956b9a2511155b79. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:43.589481 2059 consensus_queue.cc:1035] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:43.589740 847 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.589825 1185 ts_tablet_manager.cc:1905] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.589843 847 leader_election.cc:304] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.590109 710 leader_election.cc:304] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.590255 847 leader_election.cc:304] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.590344 2059 raft_consensus.cc:2798] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.590420 2059 raft_consensus.cc:491] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.590493 2059 raft_consensus.cc:3054] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.590649 2415 raft_consensus.cc:2743] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.590945 2415 raft_consensus.cc:2798] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.591004 2415 raft_consensus.cc:491] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.591056 2415 raft_consensus.cc:3054] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.593230 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:43.594122 446 catalog_manager.cc:5526] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:43.594703 2415 raft_consensus.cc:513] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.595290 2415 leader_election.cc:290] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.595650 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:43.596060 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:43.596326 2059 raft_consensus.cc:513] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.596927 2059 leader_election.cc:290] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.601778 1185 ts_tablet_manager.cc:1918] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.601888 1185 log.cc:1198] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:43.602223 1185 ts_tablet_manager.cc:1939] T aa9219fc971c4d15b0d383b20b61e540 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:43.602414 710 leader_election.cc:343] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.580724 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:59797 (request call id 70) took 797 ms (client timeout 1000 ms). Trace:
I20250112 20:51:43.603506 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9856e6b625d04f17b0869cee08f69142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.603693 906 raft_consensus.cc:3054] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:43.603226 450 rpcz_store.cc:269] 0112 20:51:42.783002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.784994 (+ 1992us) service_pool.cc:225] Handling call
0112 20:51:43.580677 (+795683us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15}
I20250112 20:51:43.606513 450 master_service.cc:946] ReplaceTablet: received request to replace tablet c23f5f085dd84d68beb277676225e7e2 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:43.606988 906 raft_consensus.cc:2463] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.607545 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7f6a1e9804e6ab822f9d4c004b1b3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.607771 910 raft_consensus.cc:2463] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.609800 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7f6a1e9804e6ab822f9d4c004b1b3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.610435 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1748fb558c4d48f49432840055d71c67
I20250112 20:51:43.611554 911 raft_consensus.cc:2463] T 2513dfbdccf9426ba98711dae084e5ef P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.615021 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet bf66492d01a64eb1b842acc279b8ee09
I20250112 20:51:43.615240 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet aa9219fc971c4d15b0d383b20b61e540 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
I20250112 20:51:43.616608 737 tablet_service.cc:1467] Processing CreateTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.616880 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1748fb558c4d48f49432840055d71c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.617015 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53518b453d224ca186fcc6b2ffab5f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.617214 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa9be85a4b94a52a28ce61a57455fd1 (DEFAULT_TABLE table=default.Table_c4fafabe80bf4a93a9c354c73ad78f81 [id=e89788942f954df2af346aa47f8fde06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.617511 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa9be85a4b94a52a28ce61a57455fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.618640 710 leader_election.cc:304] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.618911 710 leader_election.cc:304] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.619131 2208 raft_consensus.cc:2798] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.619220 2208 raft_consensus.cc:695] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.619439 2208 consensus_queue.cc:237] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.621043 745 tablet_service.cc:1467] Processing CreateTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 (DEFAULT_TABLE table=default.Table_093ad5dba68a4f0b9d96c37ac3888a83 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.621384 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.621908 738 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.622141 2059 raft_consensus.cc:2798] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.622216 2059 raft_consensus.cc:491] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.622346 2059 raft_consensus.cc:3054] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.624994 756 tablet_service.cc:1467] Processing CreateTablet for tablet f424288fe62849edbbd2d38e89331fb7 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.625300 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f424288fe62849edbbd2d38e89331fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.625663 2059 raft_consensus.cc:513] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.626395 2059 leader_election.cc:290] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.627936 710 leader_election.cc:304] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.628836 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 809) took 1682 ms. Trace:
I20250112 20:51:43.621043 754 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4261254a94a9f967a5b418fecf5d5 (DEFAULT_TABLE table=default.Table_e6d67503da2e4fd9aedb3c00d4c8e4cb [id=d73fd6d71cbc4f47b5933e51c0e2077f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.629230 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4261254a94a9f967a5b418fecf5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.631501 2059 raft_consensus.cc:2798] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.631620 2059 raft_consensus.cc:695] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.631871 2059 consensus_queue.cc:237] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.629253 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7f6a1e9804e6ab822f9d4c004b1b3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.632181 911 raft_consensus.cc:3054] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.633693 1391 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:43.635159 911 raft_consensus.cc:2463] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.635406 1186 rpcz_store.cc:276] 0112 20:51:41.945977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.946033 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:43.459103 (+1513070us) ts_tablet_manager.cc:1145] Deleting tablet 70653110aec74727acec9d63c2eb0f05
0112 20:51:43.459112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.492650 (+ 33538us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.586695 (+ 94045us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.628804 (+ 42109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512856}
I20250112 20:51:43.636611 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 806) took 1695 ms. Trace:
I20250112 20:51:43.637499 1186 rpcz_store.cc:276] 0112 20:51:41.940720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:41.954162 (+ 13442us) service_pool.cc:225] Handling call
0112 20:51:43.636449 (+1682287us) ts_tablet_manager.cc:1145] Deleting tablet d3ea347fd2694016a3cd3721b0cab608
0112 20:51:43.636466 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.636592 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651544}
I20250112 20:51:43.641366 1186 tablet_replica.cc:331] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.642066 1186 raft_consensus.cc:2238] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:43.642982 1186 pending_rounds.cc:62] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:43.643808 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1fcaaa7af46c48fbb69589e6013968c0
I20250112 20:51:43.647204 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e0fd4177f9394a5a82d4f5e02369b3b6
I20250112 20:51:43.648999 1186 pending_rounds.cc:69] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585058275561472 op_type: NO_OP noop_request { }
I20250112 20:51:43.649010 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2e8ea09f0c2c4ccea9540106b863d8a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.649706 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e8ea09f0c2c4ccea9540106b863d8a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.649896 1186 raft_consensus.cc:2883] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:43.651082 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_527edcc30df04186a34eb7b085689535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.651602 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_527edcc30df04186a34eb7b085689535 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.652004 1186 raft_consensus.cc:2267] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.653057 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7e40c8ac7b0842fb81f72ea44cda9bbd" } new_table_name: "default.Table_29bb69809f3848deaca3e824da508612" modify_external_catalogs: true
I20250112 20:51:43.654302 1186 ts_tablet_manager.cc:1905] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.654564 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_185f9bae27844411aa65dad64f12594e" } modify_external_catalogs: true
I20250112 20:51:43.654718 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_185f9bae27844411aa65dad64f12594e" } modify_external_catalogs: true
I20250112 20:51:43.656281 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_acc683b25e9a4c3f84ce1b9476963382" } new_table_name: "default.Table_dc131053ce3845268e1279e76cbf1ddd" modify_external_catalogs: true
I20250112 20:51:43.657536 710 leader_election.cc:304] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.659442 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.662542 2631 raft_consensus.cc:2798] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.665817 2631 raft_consensus.cc:695] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.666754 2631 consensus_queue.cc:237] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.667012 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e0034007793b4b039c081149df5039c3" } new_table_name: "default.Table_c8b8e3e3e6234cfd952aeb5116a142f9" modify_external_catalogs: true
I20250112 20:51:43.663309 747 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:43.663172 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:43.673038 710 leader_election.cc:343] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.662293 2563 raft_consensus.cc:491] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.681321 2563 raft_consensus.cc:513] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.682044 2563 leader_election.cc:290] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.670323 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7f6a1e9804e6ab822f9d4c004b1b3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:43.683308 983 leader_election.cc:343] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 70653110aec74727acec9d63c2eb0f05
I20250112 20:51:43.670290 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9856e6b625d04f17b0869cee08f69142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.670820 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d384628ba9b4f81b768eda4f796632b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:29 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.671137 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e1c32cab2c4b438d3ff4a7e674feeb with delete_type TABLET_DATA_DELETED (replaced by tablet 23dc8a695ccd4f88bd108304add01a4e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.684481 2447 tablet_replica.cc:331] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:43.684720 2447 raft_consensus.cc:2238] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.684895 2447 raft_consensus.cc:2267] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:43.685060 983 leader_election.cc:343] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 70653110aec74727acec9d63c2eb0f05
I20250112 20:51:43.685175 983 leader_election.cc:304] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.685482 2563 raft_consensus.cc:2743] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.685505 2447 ts_tablet_manager.cc:1905] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.671200 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eeb085e764941198ae7f653bc24757d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.671451 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ebb3c30194d9fa6a092d9bec4eadf with delete_type TABLET_DATA_DELETED (replaced by tablet f7b722b0eb5944a4b344a3833905a2b0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.671541 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6d5d425e6a493daa46900cbd8db166 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.671766 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 944b71897bcf4304bd9f7bb7adef34e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.672019 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 64887b3578284cb286ee122ab8c626c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.672463 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4261254a94a9f967a5b418fecf5d5 (DEFAULT_TABLE table=default.Table_e6d67503da2e4fd9aedb3c00d4c8e4cb [id=d73fd6d71cbc4f47b5933e51c0e2077f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.689411 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4261254a94a9f967a5b418fecf5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.690799 1185 tablet_replica.cc:331] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:43.690989 1185 raft_consensus.cc:2238] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.691126 1185 raft_consensus.cc:2267] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.672528 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 (DEFAULT_TABLE table=default.Table_093ad5dba68a4f0b9d96c37ac3888a83 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.693125 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.695514 1185 ts_tablet_manager.cc:1905] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.672504 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f424288fe62849edbbd2d38e89331fb7 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.672674 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1748fb558c4d48f49432840055d71c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.672823 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.672863 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa9be85a4b94a52a28ce61a57455fd1 (DEFAULT_TABLE table=default.Table_c4fafabe80bf4a93a9c354c73ad78f81 [id=e89788942f954df2af346aa47f8fde06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.697785 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f424288fe62849edbbd2d38e89331fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.672942 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.673048 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.698129 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa9be85a4b94a52a28ce61a57455fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.675065 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd46a422093495ba8b5b03039874a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.676512 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829e9dd5854961b5402b5710624612 with delete_type TABLET_DATA_DELETED (replaced by tablet 7623bebb539e485aa508e95a35225979) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.702967 1163 ts_tablet_manager.cc:1428] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.160s user 0.002s sys 0.000s
I20250112 20:51:43.703369 1163 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.711627 2623 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:43.712638 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 122386us
I20250112 20:51:43.713485 2521 consensus_queue.cc:1035] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.715564 1163 tablet_bootstrap.cc:654] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.716152 881 tablet_service.cc:1467] Processing CreateTablet for tablet 34608d460c1347a9895849ea77776299 (DEFAULT_TABLE table=default.Table_7ffaf1ff93694b688350c7dfb482f7fe [id=1bbb740eb24c404181f33c5095e845c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.716581 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34608d460c1347a9895849ea77776299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.719924 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:43.720202 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:43.720664 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:43.724685 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]). Expected version 1 got 0
I20250112 20:51:43.725028 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 0dc920e92de545fc961f82e7d0de28cd (table default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1]). Expected version 3 got 0
I20250112 20:51:43.725186 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 36935e6d8c5643388ad163458e2b387c (table default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51]). Expected version 1 got 0
I20250112 20:51:43.725507 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet f3facda2ee004b2e9487194ba040c9cf (table default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2]). Expected version 2 got 0
I20250112 20:51:43.725656 883 tablet_service.cc:1467] Processing CreateTablet for tablet 31c7729ffb494acb97bb2dfe356e2a0f (DEFAULT_TABLE table=default.Table_ca52e028a3b84bfe8dcb38464702a6ca [id=3f284788897545969907737ec6333966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.726081 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c7729ffb494acb97bb2dfe356e2a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.731452 1178 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.731701 1178 ts_tablet_manager.cc:1397] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.195s user 0.011s sys 0.002s
I20250112 20:51:43.732726 1178 raft_consensus.cc:357] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.736092 1178 raft_consensus.cc:383] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.736217 1178 raft_consensus.cc:738] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.736893 883 tablet_service.cc:1467] Processing CreateTablet for tablet fcd5b6e74e8040239d75ec58a9dea13f (DEFAULT_TABLE table=default.Table_1c840ad1fe9c49aba52e7a3dc7281fe4 [id=bcd83f917a9d4b5c83cd1fda20e9efb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.736673 885 tablet_service.cc:1467] Processing CreateTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.736835 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9850969d6a4a4610b5bdf5a780484094 (DEFAULT_TABLE table=default.Table_eaed25edb5a043188e95b48cdd144890 [id=84c2b483c928476c9049c20c31488daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.736990 878 tablet_service.cc:1467] Processing CreateTablet for tablet a4e4261254a94a9f967a5b418fecf5d5 (DEFAULT_TABLE table=default.Table_e6d67503da2e4fd9aedb3c00d4c8e4cb [id=d73fd6d71cbc4f47b5933e51c0e2077f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.738415 2208 raft_consensus.cc:491] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.724563 2447 ts_tablet_manager.cc:1918] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:43.738831 2447 log.cc:1198] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5d384628ba9b4f81b768eda4f796632b
I20250112 20:51:43.738719 2208 raft_consensus.cc:513] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.739496 2447 ts_tablet_manager.cc:1939] T 5d384628ba9b4f81b768eda4f796632b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:43.740330 2208 leader_election.cc:290] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.736673 892 tablet_service.cc:1467] Processing CreateTablet for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f (DEFAULT_TABLE table=default.Table_fc90d10b52794ebea4e3e19db3cc8dc8 [id=c8d1c0a4595949cf8af1f82f99f1f3c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.737740 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcd5b6e74e8040239d75ec58a9dea13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.741920 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ad7cfc9f8c4cbdadb7682eb162a15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.742115 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53518b453d224ca186fcc6b2ffab5f04. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:43.744431 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39271, user_credentials={real_user=slave}} blocked reactor thread for 69373.8us
I20250112 20:51:43.744894 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4261254a94a9f967a5b418fecf5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.745796 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3343e1d7f1d04a498bd0ee00311b6a61" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.745994 909 raft_consensus.cc:2463] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.748968 710 leader_election.cc:304] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.737131 1178 consensus_queue.cc:260] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.749262 2631 raft_consensus.cc:2798] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.749332 2631 raft_consensus.cc:491] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.749392 2631 raft_consensus.cc:3054] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.750537 2447 tablet_replica.cc:331] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:43.750726 2447 raft_consensus.cc:2238] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.750860 2447 raft_consensus.cc:2267] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.751397 2447 ts_tablet_manager.cc:1905] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.751278 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8f0fb6214ee84e8a88e5292e5db7756d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.751834 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f0fb6214ee84e8a88e5292e5db7756d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.751892 2631 raft_consensus.cc:513] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.752552 2631 leader_election.cc:290] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.753676 1178 ts_tablet_manager.cc:1428] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250112 20:51:43.753813 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8d21e147bdd0405cbc9b0580dd27b827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:43.754473 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d21e147bdd0405cbc9b0580dd27b827 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:43.760663 1178 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.762446 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9850969d6a4a4610b5bdf5a780484094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.763988 1186 ts_tablet_manager.cc:1918] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.764206 1186 log.cc:1198] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:43.765975 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3343e1d7f1d04a498bd0ee00311b6a61" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.766146 909 raft_consensus.cc:3054] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.766969 2521 raft_consensus.cc:491] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.767113 2521 raft_consensus.cc:513] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.767876 2521 leader_election.cc:290] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.768230 1178 tablet_bootstrap.cc:654] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.768429 1186 ts_tablet_manager.cc:1939] T bc05d287d81c46558acdd9a42edda6f8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.771001 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 810) took 1362 ms. Trace:
I20250112 20:51:43.771131 1186 rpcz_store.cc:276] 0112 20:51:42.408928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.408987 (+ 59us) service_pool.cc:225] Handling call
0112 20:51:43.641311 (+1232324us) ts_tablet_manager.cc:1145] Deleting tablet bc05d287d81c46558acdd9a42edda6f8
0112 20:51:43.641324 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.674436 (+ 33112us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.763900 (+ 89464us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.770984 (+ 7084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232135}
I20250112 20:51:43.771888 1186 tablet_replica.cc:331] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.772217 1186 raft_consensus.cc:2238] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.772655 1186 raft_consensus.cc:2267] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.773283 1186 ts_tablet_manager.cc:1905] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.773686 1164 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.773828 1164 ts_tablet_manager.cc:1397] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.268s user 0.001s sys 0.001s
I20250112 20:51:43.774276 909 raft_consensus.cc:2463] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.774638 710 leader_election.cc:304] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.774909 2059 raft_consensus.cc:2798] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.775167 2059 raft_consensus.cc:695] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.775425 2059 consensus_queue.cc:237] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:43.780136 1186 ts_tablet_manager.cc:1918] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.780258 1186 log.cc:1198] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:43.774488 1164 raft_consensus.cc:357] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.780594 1186 ts_tablet_manager.cc:1939] T 64e1c32cab2c4b438d3ff4a7e674feeb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.780853 1164 raft_consensus.cc:383] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.781031 1164 raft_consensus.cc:738] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.781340 1164 consensus_queue.cc:260] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.782302 1164 ts_tablet_manager.cc:1428] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:43.782240 875 tablet_service.cc:1467] Processing CreateTablet for tablet f424288fe62849edbbd2d38e89331fb7 (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.782676 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f424288fe62849edbbd2d38e89331fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.782924 1164 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.784392 1164 tablet_bootstrap.cc:654] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.787281 1185 ts_tablet_manager.cc:1918] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.787521 1185 log.cc:1198] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:43.787926 1185 ts_tablet_manager.cc:1939] T ad1a4b5ac2f94874b99b3ef790a034fd P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:43.789003 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet ad1a4b5ac2f94874b99b3ef790a034fd (table default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]) successfully deleted
I20250112 20:51:43.789373 1185 tablet_replica.cc:331] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:43.789618 1185 raft_consensus.cc:2238] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.789824 1185 pending_rounds.cc:62] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:43.789909 1185 pending_rounds.cc:69] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585058275561472 op_type: NO_OP noop_request { }
I20250112 20:51:43.790063 1185 raft_consensus.cc:2883] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:43.790210 1185 raft_consensus.cc:2267] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.790951 1185 ts_tablet_manager.cc:1905] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.793288 875 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa9be85a4b94a52a28ce61a57455fd1 (DEFAULT_TABLE table=default.Table_c4fafabe80bf4a93a9c354c73ad78f81 [id=e89788942f954df2af346aa47f8fde06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.793673 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa9be85a4b94a52a28ce61a57455fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.794711 883 tablet_service.cc:1467] Processing CreateTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 (DEFAULT_TABLE table=default.Table_093ad5dba68a4f0b9d96c37ac3888a83 [id=6d119f25a5734764bf99d4aba720286f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:43.795125 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:43.797915 1178 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.798035 1185 ts_tablet_manager.cc:1918] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.798143 1178 ts_tablet_manager.cc:1397] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20250112 20:51:43.798221 1185 log.cc:1198] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:43.798629 1185 ts_tablet_manager.cc:1939] T bc05d287d81c46558acdd9a42edda6f8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:43.799232 1178 raft_consensus.cc:357] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.799525 1178 raft_consensus.cc:383] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.799626 1178 raft_consensus.cc:738] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.799666 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet bc05d287d81c46558acdd9a42edda6f8 (table default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]) successfully deleted
I20250112 20:51:43.799925 1178 consensus_queue.cc:260] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.800175 1164 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.800293 1164 ts_tablet_manager.cc:1397] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
W20250112 20:51:43.800371 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 70653110aec74727acec9d63c2eb0f05 because the tablet was not found. No further retry: Not found: Tablet not found: 70653110aec74727acec9d63c2eb0f05
I20250112 20:51:43.800665 1178 ts_tablet_manager.cc:1428] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 20:51:43.801007 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1748fb558c4d48f49432840055d71c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1748fb558c4d48f49432840055d71c67 already in progress: creating tablet
I20250112 20:51:43.801017 878 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.801205 892 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.801213 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcaaa7af46c48fbb69589e6013968c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.801496 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.801614 892 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.800889 1164 raft_consensus.cc:357] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:43.801707 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bf66492d01a64eb1b842acc279b8ee09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf66492d01a64eb1b842acc279b8ee09 already in progress: creating tablet
I20250112 20:51:43.801726 1164 raft_consensus.cc:383] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.801793 1164 raft_consensus.cc:738] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.801966 1164 consensus_queue.cc:260] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.802057 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:43.802389 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e0fd4177f9394a5a82d4f5e02369b3b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fd4177f9394a5a82d4f5e02369b3b6 already in progress: creating tablet
I20250112 20:51:40.553464 2137 rpcz_store.cc:276] 0112 20:51:38.081290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:38.081347 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:40.040699 (+1959352us) raft_consensus.cc:1466] Updating replica for 358 ops
0112 20:51:40.074868 (+ 34169us) raft_consensus.cc:1529] Early marking committed up to index 0
0112 20:51:40.074882 (+ 14us) raft_consensus.cc:1534] Triggering prepare for 358 ops
0112 20:51:40.075680 (+ 798us) write_op.cc:270] Start()
0112 20:51:40.075745 (+ 65us) write_op.cc:276] Timestamp: P: 1736715090932532 usec, L: 0
0112 20:51:40.142383 (+ 66638us) write_op.cc:270] Start()
0112 20:51:40.142420 (+ 37us) write_op.cc:276] Timestamp: P: 1736715090934757 usec, L: 0
0112 20:51:40.142661 (+ 241us) write_op.cc:270] Start()
0112 20:51:40.142697 (+ 36us) write_op.cc:276] Timestamp: P: 1736715090945456 usec, L: 0
0112 20:51:40.142867 (+ 170us) write_op.cc:270] Start()
0112 20:51:40.142888 (+ 21us) write_op.cc:276] Timestamp: P: 1736715090946037 usec, L: 0
0112 20:51:40.143050 (+ 162us) write_op.cc:270] Start()
0112 20:51:40.143082 (+ 32us) write_op.cc:276] Timestamp: P: 1736715090947344 usec, L: 0
0112 20:51:40.143228 (+ 146us) write_op.cc:270] Start()
0112 20:51:40.143249 (+ 21us) write_op.cc:276] Timestamp: P: 1736715090948489 usec, L: 0
0112 20:51:40.143417 (+ 168us) write_op.cc:270] Start()
0112 20:51:40.143458 (+ 41us) write_op.cc:276] Timestamp: P: 1736715090961132 usec, L: 0
0112 20:51:40.143648 (+ 190us) write_op.cc:270] Start()
0112 20:51:40.143670 (+ 22us) write_op.cc:276] Timestamp: P: 1736715090961742 usec, L: 0
0112 20:51:40.143839 (+ 169us) write_op.cc:270] Start()
0112 20:51:40.143858 (+ 19us) write_op.cc:276] Timestamp: P: 1736715090970096 usec, L: 0
0112 20:51:40.143995 (+ 137us) write_op.cc:270] Start()
0112 20:51:40.144015 (+ 20us) write_op.cc:276] Timestamp: P: 1736715090970627 usec, L: 0
0112 20:51:40.144213 (+ 198us) write_op.cc:270] Start()
0112 20:51:40.144234 (+ 21us) write_op.cc:276] Timestamp: P: 1736715090990283 usec, L: 0
0112 20:51:40.144366 (+ 132us) write_op.cc:270] Start()
0112 20:51:40.144385 (+ 19us) write_op.cc:276] Timestamp: P: 1736715090995829 usec, L: 0
0112 20:51:40.144523 (+ 138us) write_op.cc:270] Start()
0112 20:51:40.144543 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091009141 usec, L: 0
0112 20:51:40.144668 (+ 125us) write_op.cc:270] Start()
0112 20:51:40.144690 (+ 22us) write_op.cc:276] Timestamp: P: 1736715091060446 usec, L: 0
0112 20:51:40.161008 (+ 16318us) write_op.cc:270] Start()
0112 20:51:40.161044 (+ 36us) write_op.cc:276] Timestamp: P: 1736715091070721 usec, L: 0
0112 20:51:40.161248 (+ 204us) write_op.cc:270] Start()
0112 20:51:40.161272 (+ 24us) write_op.cc:276] Timestamp: P: 1736715091100749 usec, L: 0
0112 20:51:40.161478 (+ 206us) write_op.cc:270] Start()
0112 20:51:40.161509 (+ 31us) write_op.cc:276] Timestamp: P: 1736715091105118 usec, L: 0
0112 20:51:40.161665 (+ 156us) write_op.cc:270] Start()
0112 20:51:40.161687 (+ 22us) write_op.cc:276] Timestamp: P: 1736715091116246 usec, L: 0
0112 20:51:40.161812 (+ 125us) write_op.cc:270] Start()
0112 20:51:40.161829 (+ 17us) write_op.cc:276] Timestamp: P: 1736715091170541 usec, L: 0
0112 20:51:40.161932 (+ 103us) write_op.cc:270] Start()
0112 20:51:40.161948 (+ 16us) write_op.cc:276] Timestamp: P: 1736715091171528 usec, L: 0
0112 20:51:40.162057 (+ 109us) write_op.cc:270] Start()
0112 20:51:40.162091 (+ 34us) write_op.cc:276] Timestamp: P: 1736715091177488 usec, L: 0
0112 20:51:40.162230 (+ 139us) write_op.cc:270] Start()
0112 20:51:40.162250 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091178815 usec, L: 0
0112 20:51:40.162394 (+ 144us) write_op.cc:270] Start()
0112 20:51:40.162416 (+ 22us) write_op.cc:276] Timestamp: P: 1736715091179475 usec, L: 0
0112 20:51:40.162627 (+ 211us) write_op.cc:270] Start()
0112 20:51:40.162646 (+ 19us) write_op.cc:276] Timestamp: P: 1736715091180075 usec, L: 0
0112 20:51:40.162795 (+ 149us) write_op.cc:270] Start()
0112 20:51:40.162813 (+ 18us) write_op.cc:276] Timestamp: P: 1736715091181049 usec, L: 0
0112 20:51:40.162934 (+ 121us) write_op.cc:270] Start()
0112 20:51:40.162951 (+ 17us) write_op.cc:276] Timestamp: P: 1736715091181771 usec, L: 0
0112 20:51:40.163058 (+ 107us) write_op.cc:270] Start()
0112 20:51:40.163091 (+ 33us) write_op.cc:276] Timestamp: P: 1736715091182665 usec, L: 0
0112 20:51:40.163201 (+ 110us) write_op.cc:270] Start()
0112 20:51:40.163217 (+ 16us) write_op.cc:276] Timestamp: P: 1736715091221299 usec, L: 0
0112 20:51:40.173544 (+ 10327us) write_op.cc:270] Start()
0112 20:51:40.173578 (+ 34us) write_op.cc:276] Timestamp: P: 1736715091246733 usec, L: 0
0112 20:51:40.173740 (+ 162us) write_op.cc:270] Start()
0112 20:51:40.173764 (+ 24us) write_op.cc:276] Timestamp: P: 1736715091277175 usec, L: 0
0112 20:51:40.173906 (+ 142us) write_op.cc:270] Start()
0112 20:51:40.173927 (+ 21us) write_op.cc:276] Timestamp: P: 1736715091332516 usec, L: 0
0112 20:51:40.174091 (+ 164us) write_op.cc:270] Start()
0112 20:51:40.174111 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091333145 usec, L: 0
0112 20:51:40.174287 (+ 176us) write_op.cc:270] Start()
0112 20:51:40.174307 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091333551 usec, L: 0
0112 20:51:40.174434 (+ 127us) write_op.cc:270] Start()
0112 20:51:40.174452 (+ 18us) write_op.cc:276] Timestamp: P: 1736715091334021 usec, L: 0
0112 20:51:40.174606 (+ 154us) write_op.cc:270] Start()
0112 20:51:40.174626 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091357969 usec, L: 0
0112 20:51:40.174754 (+ 128us) write_op.cc:270] Start()
0112 20:51:40.174790 (+ 36us) write_op.cc:276] Timestamp: P: 1736715091540201 usec, L: 0
0112 20:51:40.174940 (+ 150us) write_op.cc:270] Start()
0112 20:51:40.174970 (+ 30us) write_op.cc:276] Timestamp: P: 1736715091545616 usec, L: 0
0112 20:51:40.175128 (+ 158us) write_op.cc:270] Start()
0112 20:51:40.175149 (+ 21us) write_op.cc:276] Timestamp: P: 1736715091554277 usec, L: 0
0112 20:51:40.175290 (+ 141us) write_op.cc:270] Start()
0112 20:51:40.175312 (+ 22us) write_op.cc:276] Timestamp: P: 1736715091576357 usec, L: 0
0112 20:51:40.175447 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.175467 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091626312 usec, L: 0
0112 20:51:40.175603 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.175623 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091661727 usec, L: 0
0112 20:51:40.175767 (+ 144us) write_op.cc:270] Start()
0112 20:51:40.175788 (+ 21us) write_op.cc:276] Timestamp: P: 1736715091709713 usec, L: 0
0112 20:51:40.175924 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.175944 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091724494 usec, L: 0
0112 20:51:40.176105 (+ 161us) write_op.cc:270] Start()
0112 20:51:40.176124 (+ 19us) write_op.cc:276] Timestamp: P: 1736715091725208 usec, L: 0
0112 20:51:40.176324 (+ 200us) write_op.cc:270] Start()
0112 20:51:40.176346 (+ 22us) write_op.cc:276] Timestamp: P: 1736715091846710 usec, L: 0
0112 20:51:40.176485 (+ 139us) write_op.cc:270] Start()
0112 20:51:40.176505 (+ 20us) write_op.cc:276] Timestamp: P: 1736715091880357 usec, L: 0
0112 20:51:40.176641 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.176660 (+ 19us) write_op.cc:276] Timestamp: P: 1736715091943985 usec, L: 0
0112 20:51:40.176765 (+ 105us) write_op.cc:270] Start()
0112 20:51:40.203160 (+ 26395us) write_op.cc:276] Timestamp: P: 1736715092009082 usec, L: 0
0112 20:51:40.203403 (+ 243us) write_op.cc:270] Start()
0112 20:51:40.203440 (+ 37us) write_op.cc:276] Timestamp: P: 1736715092016807 usec, L: 0
0112 20:51:40.203635 (+ 195us) write_op.cc:270] Start()
0112 20:51:40.203656 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092020557 usec, L: 0
0112 20:51:40.203885 (+ 229us) write_op.cc:270] Start()
0112 20:51:40.203909 (+ 24us) write_op.cc:276] Timestamp: P: 1736715092022045 usec, L: 0
0112 20:51:40.204096 (+ 187us) write_op.cc:270] Start()
0112 20:51:40.209346 (+ 5250us) write_op.cc:276] Timestamp: P: 1736715092023392 usec, L: 0
0112 20:51:40.209528 (+ 182us) write_op.cc:270] Start()
0112 20:51:40.209548 (+ 20us) write_op.cc:276] Timestamp: P: 1736715092029785 usec, L: 0
0112 20:51:40.209671 (+ 123us) write_op.cc:270] Start()
0112 20:51:40.209690 (+ 19us) write_op.cc:276] Timestamp: P: 1736715092031010 usec, L: 0
0112 20:51:40.209817 (+ 127us) write_op.cc:270] Start()
0112 20:51:40.209837 (+ 20us) write_op.cc:276] Timestamp: P: 1736715092032432 usec, L: 0
0112 20:51:40.209968 (+ 131us) write_op.cc:270] Start()
0112 20:51:40.209987 (+ 19us) write_op.cc:276] Timestamp: P: 1736715092067117 usec, L: 0
0112 20:51:40.210139 (+ 152us) write_op.cc:270] Start()
0112 20:51:40.210158 (+ 19us) write_op.cc:276] Timestamp: P: 1736715092077560 usec, L: 0
0112 20:51:40.210279 (+ 121us) write_op.cc:270] Start()
0112 20:51:40.210295 (+ 16us) write_op.cc:276] Timestamp: P: 1736715092087955 usec, L: 0
0112 20:51:40.210404 (+ 109us) write_op.cc:270] Start()
0112 20:51:40.210421 (+ 17us) write_op.cc:276] Timestamp: P: 1736715092095676 usec, L: 0
0112 20:51:40.210556 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.210599 (+ 43us) write_op.cc:276] Timestamp: P: 1736715092100599 usec, L: 0
0112 20:51:40.210729 (+ 130us) write_op.cc:270] Start()
0112 20:51:40.210747 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092126364 usec, L: 0
0112 20:51:40.210897 (+ 150us) write_op.cc:270] Start()
0112 20:51:40.210914 (+ 17us) write_op.cc:276] Timestamp: P: 1736715092128878 usec, L: 0
0112 20:51:40.211037 (+ 123us) write_op.cc:270] Start()
0112 20:51:40.211053 (+ 16us) write_op.cc:276] Timestamp: P: 1736715092131466 usec, L: 0
0112 20:51:40.211188 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.211205 (+ 17us) write_op.cc:276] Timestamp: P: 1736715092134052 usec, L: 0
0112 20:51:40.211331 (+ 126us) write_op.cc:270] Start()
0112 20:51:40.211351 (+ 20us) write_op.cc:276] Timestamp: P: 1736715092134631 usec, L: 0
0112 20:51:40.211483 (+ 132us) write_op.cc:270] Start()
0112 20:51:40.211501 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092152376 usec, L: 0
0112 20:51:40.211646 (+ 145us) write_op.cc:270] Start()
0112 20:51:40.211664 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092158528 usec, L: 0
0112 20:51:40.211790 (+ 126us) write_op.cc:270] Start()
0112 20:51:40.211818 (+ 28us) write_op.cc:276] Timestamp: P: 1736715092159403 usec, L: 0
0112 20:51:40.211949 (+ 131us) write_op.cc:270] Start()
0112 20:51:40.211969 (+ 20us) write_op.cc:276] Timestamp: P: 1736715092160564 usec, L: 0
0112 20:51:40.212113 (+ 144us) write_op.cc:270] Start()
0112 20:51:40.212134 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092429391 usec, L: 0
0112 20:51:40.212271 (+ 137us) write_op.cc:270] Start()
0112 20:51:40.212292 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092469967 usec, L: 0
0112 20:51:40.212434 (+ 142us) write_op.cc:270] Start()
0112 20:51:40.212457 (+ 23us) write_op.cc:276] Timestamp: P: 1736715092485253 usec, L: 0
0112 20:51:40.212595 (+ 138us) write_op.cc:270] Start()
0112 20:51:40.212613 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092541887 usec, L: 0
0112 20:51:40.212746 (+ 133us) write_op.cc:270] Start()
0112 20:51:40.212769 (+ 23us) write_op.cc:276] Timestamp: P: 1736715092576479 usec, L: 0
0112 20:51:40.226930 (+ 14161us) write_op.cc:270] Start()
0112 20:51:40.226960 (+ 30us) write_op.cc:276] Timestamp: P: 1736715092577200 usec, L: 0
0112 20:51:40.227165 (+ 205us) write_op.cc:270] Start()
0112 20:51:40.227190 (+ 25us) write_op.cc:276] Timestamp: P: 1736715092577678 usec, L: 0
0112 20:51:40.227358 (+ 168us) write_op.cc:270] Start()
0112 20:51:40.227379 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092581278 usec, L: 0
0112 20:51:40.230122 (+ 2743us) write_op.cc:270] Start()
0112 20:51:40.230153 (+ 31us) write_op.cc:276] Timestamp: P: 1736715092589993 usec, L: 0
0112 20:51:40.230336 (+ 183us) write_op.cc:270] Start()
0112 20:51:40.230376 (+ 40us) write_op.cc:276] Timestamp: P: 1736715092591511 usec, L: 0
0112 20:51:40.230570 (+ 194us) write_op.cc:270] Start()
0112 20:51:40.230606 (+ 36us) write_op.cc:276] Timestamp: P: 1736715092592518 usec, L: 0
0112 20:51:40.230758 (+ 152us) write_op.cc:270] Start()
0112 20:51:40.230779 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092593275 usec, L: 0
0112 20:51:40.230911 (+ 132us) write_op.cc:270] Start()
0112 20:51:40.230928 (+ 17us) write_op.cc:276] Timestamp: P: 1736715092593775 usec, L: 0
0112 20:51:40.231047 (+ 119us) write_op.cc:270] Start()
0112 20:51:40.231084 (+ 37us) write_op.cc:276] Timestamp: P: 1736715092615322 usec, L: 0
0112 20:51:40.231243 (+ 159us) write_op.cc:270] Start()
0112 20:51:40.231266 (+ 23us) write_op.cc:276] Timestamp: P: 1736715092734366 usec, L: 0
0112 20:51:40.231420 (+ 154us) write_op.cc:270] Start()
0112 20:51:40.231442 (+ 22us) write_op.cc:276] Timestamp: P: 1736715092735068 usec, L: 0
0112 20:51:40.231544 (+ 102us) write_op.cc:270] Start()
0112 20:51:40.231558 (+ 14us) write_op.cc:276] Timestamp: P: 1736715092735928 usec, L: 0
0112 20:51:40.231689 (+ 131us) write_op.cc:270] Start()
0112 20:51:40.231707 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092737419 usec, L: 0
0112 20:51:40.231856 (+ 149us) write_op.cc:270] Start()
0112 20:51:40.231874 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092738276 usec, L: 0
0112 20:51:40.231994 (+ 120us) write_op.cc:270] Start()
0112 20:51:40.232011 (+ 17us) write_op.cc:276] Timestamp: P: 1736715092739987 usec, L: 0
0112 20:51:40.232148 (+ 137us) write_op.cc:270] Start()
0112 20:51:40.232172 (+ 24us) write_op.cc:276] Timestamp: P: 1736715092749960 usec, L: 0
0112 20:51:40.232314 (+ 142us) write_op.cc:270] Start()
0112 20:51:40.232332 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092750835 usec, L: 0
0112 20:51:40.232456 (+ 124us) write_op.cc:270] Start()
0112 20:51:40.232487 (+ 31us) write_op.cc:276] Timestamp: P: 1736715092753305 usec, L: 0
0112 20:51:40.232659 (+ 172us) write_op.cc:270] Start()
0112 20:51:40.232678 (+ 19us) write_op.cc:276] Timestamp: P: 1736715092772040 usec, L: 0
0112 20:51:40.240966 (+ 8288us) write_op.cc:270] Start()
0112 20:51:40.240996 (+ 30us) write_op.cc:276] Timestamp: P: 1736715092810515 usec, L: 0
0112 20:51:40.241167 (+ 171us) write_op.cc:270] Start()
0112 20:51:40.241188 (+ 21us) write_op.cc:276] Timestamp: P: 1736715092820875 usec, L: 0
0112 20:51:40.241328 (+ 140us) write_op.cc:270] Start()
0112 20:51:40.241347 (+ 19us) write_op.cc:276] Timestamp: P: 1736715092823862 usec, L: 0
0112 20:51:40.241482 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.241500 (+ 18us) write_op.cc:276] Timestamp: P: 1736715092861721 usec, L: 0
0112 20:51:40.241628 (+ 128us) write_op.cc:270] Start()
0112 20:51:40.241669 (+ 41us) write_op.cc:276] Timestamp: P: 1736715093013235 usec, L: 0
0112 20:51:40.241813 (+ 144us) write_op.cc:270] Start()
0112 20:51:40.241831 (+ 18us) write_op.cc:276] Timestamp: P: 1736715093024235 usec, L: 0
0112 20:51:40.241970 (+ 139us) write_op.cc:270] Start()
0112 20:51:40.241993 (+ 23us) write_op.cc:276] Timestamp: P: 1736715093024894 usec, L: 0
0112 20:51:40.242163 (+ 170us) write_op.cc:270] Start()
0112 20:51:40.242183 (+ 20us) write_op.cc:276] Timestamp: P: 1736715093025476 usec, L: 0
0112 20:51:40.242343 (+ 160us) write_op.cc:270] Start()
0112 20:51:40.242362 (+ 19us) write_op.cc:276] Timestamp: P: 1736715093051148 usec, L: 0
0112 20:51:40.242482 (+ 120us) write_op.cc:270] Start()
0112 20:51:40.242499 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093082352 usec, L: 0
0112 20:51:40.242616 (+ 117us) write_op.cc:270] Start()
0112 20:51:40.242641 (+ 25us) write_op.cc:276] Timestamp: P: 1736715093174425 usec, L: 0
0112 20:51:40.242809 (+ 168us) write_op.cc:270] Start()
0112 20:51:40.242828 (+ 19us) write_op.cc:276] Timestamp: P: 1736715093223617 usec, L: 0
0112 20:51:40.242947 (+ 119us) write_op.cc:270] Start()
0112 20:51:40.242965 (+ 18us) write_op.cc:276] Timestamp: P: 1736715093232583 usec, L: 0
0112 20:51:40.243077 (+ 112us) write_op.cc:270] Start()
0112 20:51:40.243094 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093241567 usec, L: 0
0112 20:51:40.243228 (+ 134us) write_op.cc:270] Start()
0112 20:51:40.243248 (+ 20us) write_op.cc:276] Timestamp: P: 1736715093255501 usec, L: 0
0112 20:51:40.243378 (+ 130us) write_op.cc:270] Start()
0112 20:51:40.243395 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093273194 usec, L: 0
0112 20:51:40.243526 (+ 131us) write_op.cc:270] Start()
0112 20:51:40.243543 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093297776 usec, L: 0
0112 20:51:40.243653 (+ 110us) write_op.cc:270] Start()
0112 20:51:40.243671 (+ 18us) write_op.cc:276] Timestamp: P: 1736715093314553 usec, L: 0
0112 20:51:40.243775 (+ 104us) write_op.cc:270] Start()
0112 20:51:40.243790 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093323561 usec, L: 0
0112 20:51:40.243898 (+ 108us) write_op.cc:270] Start()
0112 20:51:40.243915 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093396430 usec, L: 0
0112 20:51:40.244028 (+ 113us) write_op.cc:270] Start()
0112 20:51:40.244045 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093419354 usec, L: 0
0112 20:51:40.244171 (+ 126us) write_op.cc:270] Start()
0112 20:51:40.244188 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093441166 usec, L: 0
0112 20:51:40.244304 (+ 116us) write_op.cc:270] Start()
0112 20:51:40.244321 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093475375 usec, L: 0
0112 20:51:40.244471 (+ 150us) write_op.cc:270] Start()
0112 20:51:40.244487 (+ 16us) write_op.cc:276] Timestamp: P: 1736715093502065 usec, L: 0
0112 20:51:40.254606 (+ 10119us) write_op.cc:270] Start()
0112 20:51:40.254641 (+ 35us) write_op.cc:276] Timestamp: P: 1736715093555764 usec, L: 0
0112 20:51:40.254826 (+ 185us) write_op.cc:270] Start()
0112 20:51:40.254847 (+ 21us) write_op.cc:276] Timestamp: P: 1736715093598559 usec, L: 0
0112 20:51:40.254983 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.254999 (+ 16us) write_op.cc:276] Timestamp: P: 1736715093608995 usec, L: 0
0112 20:51:40.255143 (+ 144us) write_op.cc:270] Start()
0112 20:51:40.255166 (+ 23us) write_op.cc:276] Timestamp: P: 1736715093609650 usec, L: 0
0112 20:51:40.255289 (+ 123us) write_op.cc:270] Start()
0112 20:51:40.255311 (+ 22us) write_op.cc:276] Timestamp: P: 1736715093610320 usec, L: 0
0112 20:51:40.255446 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.255464 (+ 18us) write_op.cc:276] Timestamp: P: 1736715093611476 usec, L: 0
0112 20:51:40.255621 (+ 157us) write_op.cc:270] Start()
0112 20:51:40.255641 (+ 20us) write_op.cc:276] Timestamp: P: 1736715093636852 usec, L: 0
0112 20:51:40.255774 (+ 133us) write_op.cc:270] Start()
0112 20:51:40.255793 (+ 19us) write_op.cc:276] Timestamp: P: 1736715093691602 usec, L: 0
0112 20:51:40.255959 (+ 166us) write_op.cc:270] Start()
0112 20:51:40.255981 (+ 22us) write_op.cc:276] Timestamp: P: 1736715093692722 usec, L: 0
0112 20:51:40.256158 (+ 177us) write_op.cc:270] Start()
0112 20:51:40.256179 (+ 21us) write_op.cc:276] Timestamp: P: 1736715093695218 usec, L: 0
0112 20:51:40.256346 (+ 167us) write_op.cc:270] Start()
0112 20:51:40.256367 (+ 21us) write_op.cc:276] Timestamp: P: 1736715093697638 usec, L: 0
0112 20:51:40.281575 (+ 25208us) write_op.cc:270] Start()
0112 20:51:40.281627 (+ 52us) write_op.cc:276] Timestamp: P: 1736715093751024 usec, L: 0
0112 20:51:40.281848 (+ 221us) write_op.cc:270] Start()
0112 20:51:40.281869 (+ 21us) write_op.cc:276] Timestamp: P: 1736715093758587 usec, L: 0
0112 20:51:40.281998 (+ 129us) write_op.cc:270] Start()
0112 20:51:40.282035 (+ 37us) write_op.cc:276] Timestamp: P: 1736715093759645 usec, L: 0
0112 20:51:40.282152 (+ 117us) write_op.cc:270] Start()
0112 20:51:40.282167 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093760384 usec, L: 0
0112 20:51:40.282270 (+ 103us) write_op.cc:270] Start()
0112 20:51:40.282285 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093761068 usec, L: 0
0112 20:51:40.282381 (+ 96us) write_op.cc:270] Start()
0112 20:51:40.282396 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093808364 usec, L: 0
0112 20:51:40.282507 (+ 111us) write_op.cc:270] Start()
0112 20:51:40.282522 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093833428 usec, L: 0
0112 20:51:40.282624 (+ 102us) write_op.cc:270] Start()
0112 20:51:40.282639 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093834386 usec, L: 0
0112 20:51:40.282735 (+ 96us) write_op.cc:270] Start()
0112 20:51:40.282750 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093835253 usec, L: 0
0112 20:51:40.282841 (+ 91us) write_op.cc:270] Start()
0112 20:51:40.282856 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093836097 usec, L: 0
0112 20:51:40.282971 (+ 115us) write_op.cc:270] Start()
0112 20:51:40.282985 (+ 14us) write_op.cc:276] Timestamp: P: 1736715093836947 usec, L: 0
0112 20:51:40.283114 (+ 129us) write_op.cc:270] Start()
0112 20:51:40.283131 (+ 17us) write_op.cc:276] Timestamp: P: 1736715093837751 usec, L: 0
0112 20:51:40.283243 (+ 112us) write_op.cc:270] Start()
0112 20:51:40.283258 (+ 15us) write_op.cc:276] Timestamp: P: 1736715093848575 usec, L: 0
0112 20:51:40.283367 (+ 109us) write_op.cc:270] Start()
0112 20:51:40.283381 (+ 14us) write_op.cc:276] Timestamp: P: 1736715093881371 usec, L: 0
0112 20:51:40.283473 (+ 92us) write_op.cc:270] Start()
0112 20:51:40.283487 (+ 14us) write_op.cc:276] Timestamp: P: 1736715093897444 usec, L: 0
0112 20:51:40.283603 (+ 116us) write_op.cc:270] Start()
0112 20:51:40.283626 (+ 23us) write_op.cc:276] Timestamp: P: 1736715093941276 usec, L: 0
0112 20:51:40.287851 (+ 4225us) write_op.cc:270] Start()
0112 20:51:40.287881 (+ 30us) write_op.cc:276] Timestamp: P: 1736715093951689 usec, L: 0
0112 20:51:40.288000 (+ 119us) write_op.cc:270] Start()
0112 20:51:40.288018 (+ 18us) write_op.cc:276] Timestamp: P: 1736715093962286 usec, L: 0
0112 20:51:40.288136 (+ 118us) write_op.cc:270] Start()
0112 20:51:40.288165 (+ 29us) write_op.cc:276] Timestamp: P: 1736715093972249 usec, L: 0
0112 20:51:40.288283 (+ 118us) write_op.cc:270] Start()
0112 20:51:40.289129 (+ 846us) write_op.cc:276] Timestamp: P: 1736715093991060 usec, L: 0
0112 20:51:40.289348 (+ 219us) write_op.cc:270] Start()
0112 20:51:40.289375 (+ 27us) write_op.cc:276] Timestamp: P: 1736715094046631 usec, L: 0
0112 20:51:40.289526 (+ 151us) write_op.cc:270] Start()
0112 20:51:40.289547 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094087255 usec, L: 0
0112 20:51:40.289687 (+ 140us) write_op.cc:270] Start()
0112 20:51:40.289711 (+ 24us) write_op.cc:276] Timestamp: P: 1736715094104094 usec, L: 0
0112 20:51:40.289872 (+ 161us) write_op.cc:270] Start()
0112 20:51:40.289893 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094170756 usec, L: 0
0112 20:51:40.290047 (+ 154us) write_op.cc:270] Start()
0112 20:51:40.290081 (+ 34us) write_op.cc:276] Timestamp: P: 1736715094179757 usec, L: 0
0112 20:51:40.290209 (+ 128us) write_op.cc:270] Start()
0112 20:51:40.290229 (+ 20us) write_op.cc:276] Timestamp: P: 1736715094180634 usec, L: 0
0112 20:51:40.290351 (+ 122us) write_op.cc:270] Start()
0112 20:51:40.290370 (+ 19us) write_op.cc:276] Timestamp: P: 1736715094181961 usec, L: 0
0112 20:51:40.290495 (+ 125us) write_op.cc:270] Start()
0112 20:51:40.290513 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094257635 usec, L: 0
0112 20:51:40.297833 (+ 7320us) write_op.cc:270] Start()
0112 20:51:40.297865 (+ 32us) write_op.cc:276] Timestamp: P: 1736715094273905 usec, L: 0
0112 20:51:40.298030 (+ 165us) write_op.cc:270] Start()
0112 20:51:40.298050 (+ 20us) write_op.cc:276] Timestamp: P: 1736715094298259 usec, L: 0
0112 20:51:40.298196 (+ 146us) write_op.cc:270] Start()
0112 20:51:40.298225 (+ 29us) write_op.cc:276] Timestamp: P: 1736715094316800 usec, L: 0
0112 20:51:40.298351 (+ 126us) write_op.cc:270] Start()
0112 20:51:40.298369 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094328957 usec, L: 0
0112 20:51:40.298505 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.298524 (+ 19us) write_op.cc:276] Timestamp: P: 1736715094346214 usec, L: 0
0112 20:51:40.299854 (+ 1330us) write_op.cc:270] Start()
0112 20:51:40.299889 (+ 35us) write_op.cc:276] Timestamp: P: 1736715094359818 usec, L: 0
0112 20:51:40.300101 (+ 212us) write_op.cc:270] Start()
0112 20:51:40.300122 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094368448 usec, L: 0
0112 20:51:40.300254 (+ 132us) write_op.cc:270] Start()
0112 20:51:40.300273 (+ 19us) write_op.cc:276] Timestamp: P: 1736715094377777 usec, L: 0
0112 20:51:40.300420 (+ 147us) write_op.cc:270] Start()
0112 20:51:40.300438 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094386252 usec, L: 0
0112 20:51:40.300573 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.300591 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094388896 usec, L: 0
0112 20:51:40.300727 (+ 136us) write_op.cc:270] Start()
0112 20:51:40.300843 (+ 116us) write_op.cc:276] Timestamp: P: 1736715094407164 usec, L: 0
0112 20:51:40.301617 (+ 774us) write_op.cc:270] Start()
0112 20:51:40.301644 (+ 27us) write_op.cc:276] Timestamp: P: 1736715094415682 usec, L: 0
0112 20:51:40.301795 (+ 151us) write_op.cc:270] Start()
0112 20:51:40.301816 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094424510 usec, L: 0
0112 20:51:40.301961 (+ 145us) write_op.cc:270] Start()
0112 20:51:40.304265 (+ 2304us) write_op.cc:276] Timestamp: P: 1736715094427287 usec, L: 0
0112 20:51:40.304465 (+ 200us) write_op.cc:270] Start()
0112 20:51:40.304511 (+ 46us) write_op.cc:276] Timestamp: P: 1736715094437294 usec, L: 0
0112 20:51:40.304672 (+ 161us) write_op.cc:270] Start()
0112 20:51:40.304693 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094443910 usec, L: 0
0112 20:51:40.305472 (+ 779us) write_op.cc:270] Start()
0112 20:51:40.305497 (+ 25us) write_op.cc:276] Timestamp: P: 1736715094464505 usec, L: 0
0112 20:51:40.305637 (+ 140us) write_op.cc:270] Start()
0112 20:51:40.305656 (+ 19us) write_op.cc:276] Timestamp: P: 1736715094469275 usec, L: 0
0112 20:51:40.305771 (+ 115us) write_op.cc:270] Start()
0112 20:51:40.305789 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094493074 usec, L: 0
0112 20:51:40.305929 (+ 140us) write_op.cc:270] Start()
0112 20:51:40.305954 (+ 25us) write_op.cc:276] Timestamp: P: 1736715094504523 usec, L: 0
0112 20:51:40.306145 (+ 191us) write_op.cc:270] Start()
0112 20:51:40.306176 (+ 31us) write_op.cc:276] Timestamp: P: 1736715094523060 usec, L: 0
0112 20:51:40.306341 (+ 165us) write_op.cc:270] Start()
0112 20:51:40.306362 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094534081 usec, L: 0
0112 20:51:40.306526 (+ 164us) write_op.cc:270] Start()
0112 20:51:40.306549 (+ 23us) write_op.cc:276] Timestamp: P: 1736715094539359 usec, L: 0
0112 20:51:40.313659 (+ 7110us) write_op.cc:270] Start()
0112 20:51:40.313698 (+ 39us) write_op.cc:276] Timestamp: P: 1736715094630980 usec, L: 0
0112 20:51:40.313889 (+ 191us) write_op.cc:270] Start()
0112 20:51:40.313914 (+ 25us) write_op.cc:276] Timestamp: P: 1736715094639142 usec, L: 0
0112 20:51:40.314060 (+ 146us) write_op.cc:270] Start()
0112 20:51:40.314093 (+ 33us) write_op.cc:276] Timestamp: P: 1736715094641271 usec, L: 0
0112 20:51:40.314228 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.314249 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094693882 usec, L: 0
0112 20:51:40.314384 (+ 135us) write_op.cc:270] Start()
0112 20:51:40.314405 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094721413 usec, L: 0
0112 20:51:40.314548 (+ 143us) write_op.cc:270] Start()
0112 20:51:40.314569 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094738428 usec, L: 0
0112 20:51:40.314702 (+ 133us) write_op.cc:270] Start()
0112 20:51:40.314723 (+ 21us) write_op.cc:276] Timestamp: P: 1736715094759149 usec, L: 0
0112 20:51:40.314866 (+ 143us) write_op.cc:270] Start()
0112 20:51:40.316536 (+ 1670us) write_op.cc:276] Timestamp: P: 1736715094775761 usec, L: 0
0112 20:51:40.316717 (+ 181us) write_op.cc:270] Start()
0112 20:51:40.316739 (+ 22us) write_op.cc:276] Timestamp: P: 1736715094796134 usec, L: 0
0112 20:51:40.316896 (+ 157us) write_op.cc:270] Start()
0112 20:51:40.316916 (+ 20us) write_op.cc:276] Timestamp: P: 1736715094809918 usec, L: 0
0112 20:51:40.317101 (+ 185us) write_op.cc:270] Start()
0112 20:51:40.317121 (+ 20us) write_op.cc:276] Timestamp: P: 1736715094825289 usec, L: 0
0112 20:51:40.317241 (+ 120us) write_op.cc:270] Start()
0112 20:51:40.317259 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094849398 usec, L: 0
0112 20:51:40.317399 (+ 140us) write_op.cc:270] Start()
0112 20:51:40.317417 (+ 18us) write_op.cc:276] Timestamp: P: 1736715094859131 usec, L: 0
0112 20:51:40.317527 (+ 110us) write_op.cc:270] Start()
0112 20:51:40.317544 (+ 17us) write_op.cc:276] Timestamp: P: 1736715094870119 usec, L: 0
0112 20:51:40.317656 (+ 112us) write_op.cc:270] Start()
0112 20:51:40.317671 (+ 15us) write_op.cc:276] Timestamp: P: 1736715094879287 usec, L: 0
0112 20:51:40.317769 (+ 98us) write_op.cc:270] Start()
0112 20:51:40.317786 (+ 17us) write_op.cc:276] Timestamp: P: 1736715094887843 usec, L: 0
0112 20:51:40.317893 (+ 107us) write_op.cc:270] Start()
0112 20:51:40.317909 (+ 16us) write_op.cc:276] Timestamp: P: 1736715094924546 usec, L: 0
0112 20:51:40.318041 (+ 132us) write_op.cc:270] Start()
0112 20:51:40.318058 (+ 17us) write_op.cc:276] Timestamp: P: 1736715
I20250112 20:51:43.801028 1178 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.803068 1164 ts_tablet_manager.cc:1428] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:43.804334 1178 tablet_bootstrap.cc:654] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.805956 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1fcaaa7af46c48fbb69589e6013968c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fcaaa7af46c48fbb69589e6013968c0 already in progress: creating tablet
I20250112 20:51:43.806658 1164 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.809095 2633 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
I20250112 20:51:43.809365 2633 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:43.810825 1164 tablet_bootstrap.cc:654] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:43.813122 710 leader_election.cc:343] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.813498 710 leader_election.cc:343] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.813803 710 leader_election.cc:343] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.815794 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5d384628ba9b4f81b768eda4f796632b (table default.Table_30306a177be64cc2b4c2a1ed3351b4de [id=deeef4f847544261abe6abb65247a23a]) successfully deleted
W20250112 20:51:43.816205 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 64e1c32cab2c4b438d3ff4a7e674feeb because the tablet was not found. No further retry: Not found: Tablet not found: 64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:43.817644 2636 raft_consensus.cc:491] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.818001 2636 raft_consensus.cc:513] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.819128 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1748fb558c4d48f49432840055d71c67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.823406 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.824532 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.825304 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc920e92de545fc961f82e7d0de28cd" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.825685 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:43.825956 983 leader_election.cc:304] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.826189 2563 raft_consensus.cc:2743] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.826682 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dcd1a120684446bbe19303fb8416ef with delete_type TABLET_DATA_DELETED (replaced by tablet 717aaf9875ef4abe92ec555f71ee85d6) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.824792 2636 leader_election.cc:290] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:43.826841 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9b7d9218094a63ad5bad03f0d27eb5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.827670 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2469127891524be5ac3dc13fbcf35c7a with delete_type TABLET_DATA_DELETED (replaced by tablet 22f6b2c4ea5f4167896c78b1c64cbb62) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.827852 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d570daec845f40778c48ff3f2e2e30cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.829243 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ff8ac6f4d94c10b03a764bbf910c44 with delete_type TABLET_DATA_DELETED (replaced by tablet 691e2c3f72ab4f439678d166b9b31661) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.829391 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 70653110aec74727acec9d63c2eb0f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.829505 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbb679585fc44699553fe85ea044566 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.829613 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f34aa32d6474561988b2cf3d0e8fd3b with delete_type TABLET_DATA_DELETED (replaced by tablet 549d92191a594788bf639cd60a6b5142) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:43.830857 2447 ts_tablet_manager.cc:1918] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:43.830969 2447 log.cc:1198] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3eeb085e764941198ae7f653bc24757d
I20250112 20:51:43.830909 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3343e1d7f1d04a498bd0ee00311b6a61" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.831283 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3343e1d7f1d04a498bd0ee00311b6a61" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:43.831703 710 consensus_peers.cc:487] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:43.831888 710 leader_election.cc:343] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.832230 710 leader_election.cc:343] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.832398 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 70653110aec74727acec9d63c2eb0f05 (table default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]) successfully deleted
I20250112 20:51:43.832736 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9b7d9218094a63ad5bad03f0d27eb5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:43.832924 776 raft_consensus.cc:3054] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.834626 2447 ts_tablet_manager.cc:1939] T 3eeb085e764941198ae7f653bc24757d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:43.835570 2447 tablet_replica.cc:331] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:43.835532 776 raft_consensus.cc:2463] T 5f9b7d9218094a63ad5bad03f0d27eb5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:43.835940 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d3ea347fd2694016a3cd3721b0cab608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3ea347fd2694016a3cd3721b0cab608 already in progress: creating tablet
I20250112 20:51:43.838377 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 812) took 1429 ms. Trace:
I20250112 20:51:43.838477 1186 rpcz_store.cc:276] 0112 20:51:42.409164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.409213 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:43.771803 (+1362590us) ts_tablet_manager.cc:1145] Deleting tablet 64e1c32cab2c4b438d3ff4a7e674feeb
0112 20:51:43.771835 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.776517 (+ 4682us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.780044 (+ 3527us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.838357 (+ 58313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362428}
I20250112 20:51:43.838536 1163 tablet_bootstrap.cc:492] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.838793 1186 tablet_replica.cc:331] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.838971 1186 raft_consensus.cc:2238] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:43.839314 1186 raft_consensus.cc:2267] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.841748 1186 ts_tablet_manager.cc:1905] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.843832 1163 ts_tablet_manager.cc:1397] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.003s
I20250112 20:51:43.846479 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3eeb085e764941198ae7f653bc24757d (table default.Table_133503fd7b744d8f8e3db0843e4ac9e2 [id=dd1c1538fe8b493ba0530bfb0268c95d]) successfully deleted
I20250112 20:51:43.849435 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet bc05d287d81c46558acdd9a42edda6f8 (table default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]) successfully deleted
I20250112 20:51:43.849471 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:43.850410 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.851560 1163 raft_consensus.cc:357] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:43.852895 847 leader_election.cc:343] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:43.853214 847 consensus_peers.cc:487] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:43.853423 847 consensus_peers.cc:487] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:43.853571 847 leader_election.cc:343] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.853809 847 leader_election.cc:304] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.856947 2415 raft_consensus.cc:2798] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.857571 2108 consensus_queue.cc:1035] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:43.859393 1164 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.859596 1164 ts_tablet_manager.cc:1397] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.002s
I20250112 20:51:43.865582 1164 raft_consensus.cc:357] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.865796 1164 raft_consensus.cc:383] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.865875 1164 raft_consensus.cc:738] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.866132 1164 consensus_queue.cc:260] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.866694 1164 ts_tablet_manager.cc:1428] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:43.867053 1164 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:43.851811 2447 raft_consensus.cc:2238] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.872993 2447 raft_consensus.cc:2267] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.873935 2447 ts_tablet_manager.cc:1905] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.874485 1164 tablet_bootstrap.cc:654] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.852030 1163 raft_consensus.cc:383] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.879860 1163 raft_consensus.cc:738] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.880364 1163 consensus_queue.cc:260] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.884635 1163 ts_tablet_manager.cc:1428] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20250112 20:51:43.886391 1163 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.889614 1178 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.889773 1178 ts_tablet_manager.cc:1397] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.089s user 0.021s sys 0.001s
I20250112 20:51:43.890630 2415 raft_consensus.cc:695] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:43.891055 2415 consensus_queue.cc:237] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.890583 1178 raft_consensus.cc:357] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.892455 1178 raft_consensus.cc:383] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.892567 1178 raft_consensus.cc:738] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.892838 1178 consensus_queue.cc:260] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.893517 1178 ts_tablet_manager.cc:1428] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:43.893831 1178 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.895980 1178 tablet_bootstrap.cc:654] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.914047 1163 tablet_bootstrap.cc:654] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.916896 1178 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.917119 1178 ts_tablet_manager.cc:1397] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250112 20:51:43.917858 1178 raft_consensus.cc:357] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.919571 1178 raft_consensus.cc:383] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.919859 1178 raft_consensus.cc:738] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.920275 1178 consensus_queue.cc:260] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.921566 1178 ts_tablet_manager.cc:1428] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:43.922838 1178 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.925222 1178 tablet_bootstrap.cc:654] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.927199 2447 ts_tablet_manager.cc:1918] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:43.926503 1186 ts_tablet_manager.cc:1918] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:43.927690 1186 log.cc:1198] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ad1a4b5ac2f94874b99b3ef790a034fd
W20250112 20:51:43.926748 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 72409.9us
I20250112 20:51:43.928139 1186 ts_tablet_manager.cc:1939] T ad1a4b5ac2f94874b99b3ef790a034fd P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:43.929018 1164 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.929262 1164 ts_tablet_manager.cc:1397] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.007s
I20250112 20:51:43.930765 1164 raft_consensus.cc:357] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.931043 1164 raft_consensus.cc:383] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.931124 1164 raft_consensus.cc:738] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.931356 1164 consensus_queue.cc:260] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.931898 1164 ts_tablet_manager.cc:1428] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:43.932237 1164 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:43.928232 847 leader_election.cc:343] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.932421 847 leader_election.cc:304] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.932962 2108 raft_consensus.cc:2743] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.933632 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]) successfully deleted
W20250112 20:51:43.940346 847 consensus_peers.cc:487] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:43.940654 847 consensus_peers.cc:487] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:43.941607 2208 raft_consensus.cc:491] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.962220 2208 raft_consensus.cc:513] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.963120 2208 leader_election.cc:290] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.963372 1164 tablet_bootstrap.cc:654] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.964057 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0018e1eabf24241a358a7e00b6b5907" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:43.964255 909 raft_consensus.cc:2463] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:43.964641 710 leader_election.cc:304] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:43.964771 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 811) took 1555 ms. Trace:
I20250112 20:51:43.965118 2208 raft_consensus.cc:2798] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:43.965215 2208 raft_consensus.cc:491] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:43.965294 2208 raft_consensus.cc:3054] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:43.964898 1186 rpcz_store.cc:276] 0112 20:51:42.409058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.409442 (+ 384us) service_pool.cc:225] Handling call
0112 20:51:43.838747 (+1429305us) ts_tablet_manager.cc:1145] Deleting tablet ad1a4b5ac2f94874b99b3ef790a034fd
0112 20:51:43.838757 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:43.877485 (+ 38728us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.926402 (+ 48917us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:43.964750 (+ 38348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429180}
I20250112 20:51:43.966132 1186 tablet_replica.cc:331] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:43.966265 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ad1a4b5ac2f94874b99b3ef790a034fd (table default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]) successfully deleted
I20250112 20:51:43.966336 1186 raft_consensus.cc:2238] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.966454 1186 pending_rounds.cc:62] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:43.966517 1186 pending_rounds.cc:69] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585060025438208 op_type: NO_OP noop_request { }
I20250112 20:51:43.966653 1186 raft_consensus.cc:2883] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:43.966753 1186 raft_consensus.cc:2267] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.968636 2208 raft_consensus.cc:513] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.969103 2447 log.cc:1198] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/267ebb3c30194d9fa6a092d9bec4eadf
I20250112 20:51:43.964221 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0018e1eabf24241a358a7e00b6b5907" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:43.957101 1163 tablet_bootstrap.cc:492] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:43.970000 1163 ts_tablet_manager.cc:1397] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.001s
W20250112 20:51:43.970135 710 leader_election.cc:343] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.971010 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0018e1eabf24241a358a7e00b6b5907" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:43.971253 2208 leader_election.cc:290] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:43.971516 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0018e1eabf24241a358a7e00b6b5907" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:43.971684 909 raft_consensus.cc:3054] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:43.971917 710 leader_election.cc:343] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:43.972252 1163 raft_consensus.cc:357] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.972505 1163 raft_consensus.cc:383] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.972595 1163 raft_consensus.cc:738] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.972890 1163 consensus_queue.cc:260] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.974462 1163 ts_tablet_manager.cc:1428] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:43.974550 909 raft_consensus.cc:2463] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:43.977036 1163 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:43.977317 710 leader_election.cc:304] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:43.973528 2447 ts_tablet_manager.cc:1939] T 267ebb3c30194d9fa6a092d9bec4eadf P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:43.980947 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 267ebb3c30194d9fa6a092d9bec4eadf (table default.Table_41daf866320a48988a8961ef716b8cf0 [id=835b8c82c0fa44999e94e3e73ff6da77]) successfully deleted
I20250112 20:51:43.982065 2636 raft_consensus.cc:491] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:43.982227 2636 raft_consensus.cc:513] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.983004 2636 leader_election.cc:290] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:43.984063 983 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 8b7b788acf8047acb34ca2596bbf141b
W20250112 20:51:43.984424 983 leader_election.cc:343] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 8b7b788acf8047acb34ca2596bbf141b
I20250112 20:51:43.984508 983 leader_election.cc:304] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:43.984889 2636 raft_consensus.cc:2743] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:43.985225 2447 tablet_replica.cc:331] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:43.985424 1186 ts_tablet_manager.cc:1905] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.988898 2208 raft_consensus.cc:2798] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:43.989432 2208 raft_consensus.cc:695] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:43.989792 2208 consensus_queue.cc:237] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.989980 1178 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:43.990096 1178 ts_tablet_manager.cc:1397] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.008s
I20250112 20:51:43.990290 1163 tablet_bootstrap.cc:654] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.990883 1178 raft_consensus.cc:357] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.991070 1178 raft_consensus.cc:383] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:43.991119 1178 raft_consensus.cc:738] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:43.991189 2447 raft_consensus.cc:2238] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:43.991393 2447 raft_consensus.cc:2267] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:43.991528 1178 consensus_queue.cc:260] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:43.992111 1178 ts_tablet_manager.cc:1428] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:43.992169 2447 ts_tablet_manager.cc:1905] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:43.992477 1178 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:43.994163 1178 tablet_bootstrap.cc:654] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:43.997582 1164 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:43.997735 1164 ts_tablet_manager.cc:1397] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
I20250112 20:51:44.001174 2108 raft_consensus.cc:491] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.001415 2108 raft_consensus.cc:513] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.003742 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.003747 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu pid 1218: Timed out: CheckLeaks RPC to 127.0.104.61:39271 timed out after 1.000s (SENT)
I20250112 20:51:44.003908 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu with pid 1218
W20250112 20:51:44.005473 847 leader_election.cc:343] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.005937 2108 leader_election.cc:290] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.006422 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.006515 1178 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
W20250112 20:51:44.006984 847 leader_election.cc:343] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.007419 847 leader_election.cc:304] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.007802 2521 raft_consensus.cc:2743] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.008570 1178 ts_tablet_manager.cc:1397] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250112 20:51:44.010056 1178 raft_consensus.cc:357] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.010358 1178 raft_consensus.cc:383] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.010470 1178 raft_consensus.cc:738] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.010783 1178 consensus_queue.cc:260] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.011512 1178 ts_tablet_manager.cc:1428] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:44.011881 1178 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.014746 1163 tablet_bootstrap.cc:492] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.017241 1163 ts_tablet_manager.cc:1397] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250112 20:51:43.999184 1164 raft_consensus.cc:357] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.017596 1178 tablet_bootstrap.cc:654] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.017827 1164 raft_consensus.cc:383] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.019941 1163 raft_consensus.cc:357] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.020293 1164 raft_consensus.cc:738] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.020434 1178 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.020545 1178 ts_tablet_manager.cc:1397] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20250112 20:51:44.020900 1164 consensus_queue.cc:260] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.021211 1178 raft_consensus.cc:357] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.021399 1178 raft_consensus.cc:383] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.021463 1178 raft_consensus.cc:738] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.021662 1178 consensus_queue.cc:260] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.022276 1178 ts_tablet_manager.cc:1428] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:44.022711 1163 raft_consensus.cc:383] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.023149 1178 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.023313 1163 raft_consensus.cc:738] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.022325 1164 ts_tablet_manager.cc:1428] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.025s user 0.002s sys 0.001s
I20250112 20:51:44.024541 1164 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.025657 1178 tablet_bootstrap.cc:654] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.025899 1163 consensus_queue.cc:260] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.024884 2208 raft_consensus.cc:491] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.028913 2208 raft_consensus.cc:513] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.030301 2208 leader_election.cc:290] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.030560 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.029819 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.031284 710 leader_election.cc:343] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.031334 1163 ts_tablet_manager.cc:1428] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
W20250112 20:51:44.031476 710 leader_election.cc:343] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.031545 710 leader_election.cc:304] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:44.031585 1163 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.031875 2059 raft_consensus.cc:2743] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.033120 1163 tablet_bootstrap.cc:654] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.034651 1164 tablet_bootstrap.cc:654] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.043048 2623 tablet.cc:1721] T f3facda2ee004b2e9487194ba040c9cf P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:44.044598 1178 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.044731 1178 ts_tablet_manager.cc:1397] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20250112 20:51:44.045114 1186 ts_tablet_manager.cc:1918] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.045225 1186 log.cc:1198] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:44.045481 1178 raft_consensus.cc:357] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.045671 1178 raft_consensus.cc:383] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.045719 1178 raft_consensus.cc:738] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.045921 1178 consensus_queue.cc:260] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.046406 1178 ts_tablet_manager.cc:1428] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:44.046648 1178 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.048252 1178 tablet_bootstrap.cc:654] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.045603 1186 ts_tablet_manager.cc:1939] T aa9219fc971c4d15b0d383b20b61e540 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.052013 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 818) took 1515 ms. Trace:
I20250112 20:51:44.052150 1186 rpcz_store.cc:276] 0112 20:51:42.536703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.536773 (+ 70us) service_pool.cc:225] Handling call
0112 20:51:43.966080 (+1429307us) ts_tablet_manager.cc:1145] Deleting tablet aa9219fc971c4d15b0d383b20b61e540
0112 20:51:43.966092 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.018297 (+ 52205us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.028726 (+ 10429us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.051994 (+ 23268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429108}
I20250112 20:51:44.052683 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet aa9219fc971c4d15b0d383b20b61e540 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
I20250112 20:51:44.053148 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 821) took 1378 ms. Trace:
I20250112 20:51:44.053263 1186 rpcz_store.cc:276] 0112 20:51:42.674895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.674946 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:44.053013 (+1378067us) ts_tablet_manager.cc:1145] Deleting tablet fae080f8e4b7436b9166c85604a77dfa
0112 20:51:44.053025 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.053131 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376978}
W20250112 20:51:44.053884 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet fae080f8e4b7436b9166c85604a77dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fae080f8e4b7436b9166c85604a77dfa already in progress: creating tablet
I20250112 20:51:44.054302 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 822) took 1378 ms. Trace:
I20250112 20:51:44.054409 1186 rpcz_store.cc:276] 0112 20:51:42.675749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.675802 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:44.054186 (+1378384us) ts_tablet_manager.cc:1145] Deleting tablet 5f264d84ae4846d1afaaabc41d5789f4
0112 20:51:44.054197 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.054288 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377226}
W20250112 20:51:44.055033 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5f264d84ae4846d1afaaabc41d5789f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f264d84ae4846d1afaaabc41d5789f4 already in progress: creating tablet
I20250112 20:51:44.055425 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 820) took 1381 ms. Trace:
I20250112 20:51:44.055634 1186 rpcz_store.cc:276] 0112 20:51:42.673917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.675639 (+ 1722us) service_pool.cc:225] Handling call
0112 20:51:44.055322 (+1379683us) ts_tablet_manager.cc:1145] Deleting tablet f9055531da4c453fa722eb7c78fa0e78
0112 20:51:44.055333 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.055409 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373991}
W20250112 20:51:44.056385 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f9055531da4c453fa722eb7c78fa0e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9055531da4c453fa722eb7c78fa0e78 already in progress: creating tablet
I20250112 20:51:44.037849 2447 ts_tablet_manager.cc:1918] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.043726 2613 tablet.cc:1721] T f3facda2ee004b2e9487194ba040c9cf P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:44.058061 2447 log.cc:1198] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9c6d5d425e6a493daa46900cbd8db166
I20250112 20:51:44.058430 2447 ts_tablet_manager.cc:1939] T 9c6d5d425e6a493daa46900cbd8db166 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.059783 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9c6d5d425e6a493daa46900cbd8db166 (table default.Table_a9245c5997d7416fa75ebf6202856645 [id=942d944d07b146fbb1e30e593fa0a266]) successfully deleted
W20250112 20:51:44.061395 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fb09123a64e744c89cb8343fc27d8d60 consensus state has no leader
I20250112 20:51:44.062017 2676 tablet.cc:1721] T f3facda2ee004b2e9487194ba040c9cf P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:44.064302 712 connection.cc:537] client connection to 127.0.104.61:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:44.065438 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:39271 (error 108) [suppressed 1 similar messages]
I20250112 20:51:44.079017 1164 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.059481 2447 tablet_replica.cc:331] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:44.063895 985 connection.cc:537] client connection to 127.0.104.61:39271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 20:51:44.085354 1163 tablet_bootstrap.cc:492] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.083163 2208 raft_consensus.cc:491] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.085827 2208 raft_consensus.cc:513] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.085978 1178 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.086140 1178 ts_tablet_manager.cc:1397] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.010s
W20250112 20:51:44.086407 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271): Couldn't send request to peer b985fd656ce44131bc91ccfeb79587c5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.086474 2208 leader_election.cc:290] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.087154 1178 raft_consensus.cc:357] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.087337 1178 raft_consensus.cc:383] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.087394 1178 raft_consensus.cc:738] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.087589 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7117edf078848c0a1c83ab8e804dfed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.087622 1178 consensus_queue.cc:260] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.087821 909 raft_consensus.cc:2463] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.088176 1178 ts_tablet_manager.cc:1428] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:44.088454 1178 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.088603 1164 ts_tablet_manager.cc:1397] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.001s
I20250112 20:51:44.089012 710 leader_election.cc:304] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.089236 2059 raft_consensus.cc:2798] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.089318 2059 raft_consensus.cc:491] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.089390 2059 raft_consensus.cc:3054] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.089368 1163 ts_tablet_manager.cc:1397] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20250112 20:51:44.090148 1178 tablet_bootstrap.cc:654] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.090059 1163 raft_consensus.cc:357] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.090224 1163 raft_consensus.cc:383] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.090283 1163 raft_consensus.cc:738] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.091073 1164 raft_consensus.cc:357] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.091274 1164 raft_consensus.cc:383] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.091329 1164 raft_consensus.cc:738] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.091499 1164 consensus_queue.cc:260] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.091904 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 823) took 1415 ms. Trace:
I20250112 20:51:44.091977 1186 rpcz_store.cc:276] 0112 20:51:42.676680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.676722 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:44.056697 (+1379975us) ts_tablet_manager.cc:1145] Deleting tablet 36f95046c87846b2aa7e15430c1480b5
0112 20:51:44.056707 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.091884 (+ 35177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361368}
I20250112 20:51:44.092379 1186 tablet_replica.cc:331] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:44.092481 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 36f95046c87846b2aa7e15430c1480b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f95046c87846b2aa7e15430c1480b5 already in progress: creating tablet
I20250112 20:51:44.092566 1186 raft_consensus.cc:2238] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.092868 1186 raft_consensus.cc:2267] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.093374 1186 ts_tablet_manager.cc:1905] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:44.093533 710 consensus_peers.cc:487] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.094036 2623 tablet.cc:1721] T 0dc920e92de545fc961f82e7d0de28cd P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.094514 2676 tablet.cc:1721] T 0dc920e92de545fc961f82e7d0de28cd P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.093878 2059 raft_consensus.cc:513] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.085650 2447 raft_consensus.cc:2238] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.085826 2636 raft_consensus.cc:491] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.096489 2636 raft_consensus.cc:513] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.096930 1163 consensus_queue.cc:260] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.097177 2636 leader_election.cc:290] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.097198 1178 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.097301 1178 ts_tablet_manager.cc:1397] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250112 20:51:44.097640 1163 ts_tablet_manager.cc:1428] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250112 20:51:44.098021 1163 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.088191 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7117edf078848c0a1c83ab8e804dfed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.098065 1178 raft_consensus.cc:357] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.098234 1178 raft_consensus.cc:383] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.098280 1178 raft_consensus.cc:738] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:44.098630 710 leader_election.cc:343] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.099742 1163 tablet_bootstrap.cc:654] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.100092 1178 consensus_queue.cc:260] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.100518 2059 leader_election.cc:290] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.100579 1178 ts_tablet_manager.cc:1428] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:44.100878 1178 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.100847 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.100945 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7117edf078848c0a1c83ab8e804dfed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.101092 909 raft_consensus.cc:3054] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.102603 1164 ts_tablet_manager.cc:1428] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 20:51:44.102874 1164 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.104758 909 raft_consensus.cc:2463] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.104764 1178 tablet_bootstrap.cc:654] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.105438 710 leader_election.cc:304] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.109143 2059 raft_consensus.cc:2798] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.109282 2059 raft_consensus.cc:695] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.109894 2059 consensus_queue.cc:237] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.111035 847 consensus_peers.cc:487] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.112455 1186 ts_tablet_manager.cc:1918] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:44.112562 1186 log.cc:1198] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8c697bba271f4455956b9a2511155b79
I20250112 20:51:44.113039 1178 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.113168 1178 ts_tablet_manager.cc:1397] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20250112 20:51:44.113440 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7117edf078848c0a1c83ab8e804dfed" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.113771 1178 raft_consensus.cc:357] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:44.113901 710 leader_election.cc:343] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.113943 1178 raft_consensus.cc:383] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.113996 1178 raft_consensus.cc:738] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.114413 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2111e988ea634e999a1bbbf7b270153b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.115190 1186 ts_tablet_manager.cc:1939] T 8c697bba271f4455956b9a2511155b79 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.115926 1164 tablet_bootstrap.cc:654] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.118172 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 824) took 1406 ms. Trace:
I20250112 20:51:44.118245 1186 rpcz_store.cc:276] 0112 20:51:42.711921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.713062 (+ 1141us) service_pool.cc:225] Handling call
0112 20:51:44.092330 (+1379268us) ts_tablet_manager.cc:1145] Deleting tablet 8c697bba271f4455956b9a2511155b79
0112 20:51:44.092340 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.096528 (+ 4188us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.112349 (+ 15821us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.118156 (+ 5807us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379049}
I20250112 20:51:44.119658 2563 raft_consensus.cc:491] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.119699 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 825) took 1401 ms. Trace:
I20250112 20:51:44.119760 1186 rpcz_store.cc:276] 0112 20:51:42.717916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.717973 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:44.119570 (+1401597us) ts_tablet_manager.cc:1145] Deleting tablet 2440e0bdf8ad4ab7805c892e28834acd
0112 20:51:44.119579 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.119684 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401379}
I20250112 20:51:44.119792 2563 raft_consensus.cc:513] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.120098 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 827) took 1336 ms. Trace:
I20250112 20:51:44.119765 1178 consensus_queue.cc:260] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.120164 1186 rpcz_store.cc:276] 0112 20:51:42.784117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.784182 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:44.119994 (+1335812us) ts_tablet_manager.cc:1145] Deleting tablet 0a87da1ea4054ffc81b9c1c792f0cc5d
0112 20:51:44.120003 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.120083 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335609}
I20250112 20:51:44.120402 1186 tablet_replica.cc:331] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.120417 2563 leader_election.cc:290] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.120608 1186 raft_consensus.cc:2238] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.120782 1186 raft_consensus.cc:2267] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.120946 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8c697bba271f4455956b9a2511155b79 (table default.Table_6c4fb46bcd8f49ae96cf3ae55ffd319f [id=fc1d61ae0aec4a519360241bfcae3281]) successfully deleted
W20250112 20:51:44.121270 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2440e0bdf8ad4ab7805c892e28834acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2440e0bdf8ad4ab7805c892e28834acd already in progress: creating tablet
W20250112 20:51:44.121482 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 0a87da1ea4054ffc81b9c1c792f0cc5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a87da1ea4054ffc81b9c1c792f0cc5d already in progress: creating tablet
I20250112 20:51:44.122593 1178 ts_tablet_manager.cc:1428] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250112 20:51:44.122889 1178 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.127261 1178 tablet_bootstrap.cc:654] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.128360 1186 ts_tablet_manager.cc:1905] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:44.134754 983 leader_election.cc:343] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:44.135269 1163 tablet_bootstrap.cc:492] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.135406 1163 ts_tablet_manager.cc:1397] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250112 20:51:44.136176 1186 ts_tablet_manager.cc:1918] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.136305 1186 log.cc:1198] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:44.136351 1163 raft_consensus.cc:357] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.136535 1163 raft_consensus.cc:383] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.136613 1163 raft_consensus.cc:738] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.136662 1186 ts_tablet_manager.cc:1939] T 6092a31de9bc47e4bae5f5aebb180c6d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.136821 1163 consensus_queue.cc:260] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.137576 1163 ts_tablet_manager.cc:1428] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:44.137898 1163 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.140008 1163 tablet_bootstrap.cc:654] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:44.142863 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f6ad2ac50c745448031bdf94e0eea59 consensus state has no leader
I20250112 20:51:44.143194 455 master_service.cc:946] ReplaceTablet: received request to replace tablet c45ac747ab7f4990a03a39553424e951 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:44.143568 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cf6d73ab981c4e87aa229a49fb3b6913
I20250112 20:51:44.144088 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 9498a04f20dd4cf4a283f17d37ca49ec from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:44.144443 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 940756f8867b4cde91fb0e89e6069265 consensus state has no leader
I20250112 20:51:44.144651 453 master_service.cc:946] ReplaceTablet: received request to replace tablet d6a0679bf1ed4bbc85f14f62dfcc114d from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:44.145956 883 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.146252 741 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:44.147069 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cf6d73ab981c4e87aa229a49fb3b6913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d73ab981c4e87aa229a49fb3b6913 already in progress: creating tablet
I20250112 20:51:44.153712 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 829) took 1231 ms. Trace:
I20250112 20:51:44.155268 1186 rpcz_store.cc:276] 0112 20:51:42.922714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.924825 (+ 2111us) service_pool.cc:225] Handling call
0112 20:51:44.120365 (+1195540us) ts_tablet_manager.cc:1145] Deleting tablet 6092a31de9bc47e4bae5f5aebb180c6d
0112 20:51:44.120373 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.133256 (+ 12883us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.136089 (+ 2833us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.153681 (+ 17592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195371}
I20250112 20:51:44.155937 2521 raft_consensus.cc:491] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.156055 2521 raft_consensus.cc:513] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.156566 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6d73ab981c4e87aa229a49fb3b6913 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.156730 2521 leader_election.cc:290] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:44.160733 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.161403 847 leader_election.cc:343] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.161823 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 6092a31de9bc47e4bae5f5aebb180c6d (table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]) successfully deleted
I20250112 20:51:44.162532 1186 tablet_replica.cc:331] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.163086 1186 raft_consensus.cc:2238] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.163872 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:44.164403 847 leader_election.cc:343] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.164520 847 leader_election.cc:304] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.164888 2521 raft_consensus.cc:2743] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.166304 1186 pending_rounds.cc:62] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:44.166597 1186 pending_rounds.cc:69] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585062326288384 op_type: NO_OP noop_request { }
I20250112 20:51:44.167637 983 leader_election.cc:304] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.168170 2563 raft_consensus.cc:2743] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.168414 1163 tablet_bootstrap.cc:492] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.168565 1163 ts_tablet_manager.cc:1397] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20250112 20:51:44.168754 1049 raft_consensus.cc:3054] T 2111e988ea634e999a1bbbf7b270153b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.173964 2632 consensus_queue.cc:1035] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.174624 2447 raft_consensus.cc:2267] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.179847 1186 raft_consensus.cc:2883] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:44.180999 2447 ts_tablet_manager.cc:1905] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.180667 1163 raft_consensus.cc:357] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.181264 1163 raft_consensus.cc:383] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.181336 1163 raft_consensus.cc:738] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.181555 1163 consensus_queue.cc:260] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.182543 983 leader_election.cc:343] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:44.182646 983 leader_election.cc:304] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.181947 1178 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.182884 2563 raft_consensus.cc:2743] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.182826 1186 raft_consensus.cc:2267] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.182859 1178 ts_tablet_manager.cc:1397] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.022s
I20250112 20:51:44.184911 1178 raft_consensus.cc:357] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.185139 1178 raft_consensus.cc:383] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.185218 1178 raft_consensus.cc:738] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.185842 1178 consensus_queue.cc:260] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.186729 1178 ts_tablet_manager.cc:1428] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:44.187000 1178 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.188617 1178 tablet_bootstrap.cc:654] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.189261 1186 ts_tablet_manager.cc:1905] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.190665 1163 ts_tablet_manager.cc:1428] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250112 20:51:44.191088 1163 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.193850 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c23f5f085dd84d68beb277676225e7e2
I20250112 20:51:44.194404 450 catalog_manager.cc:6596] ReplaceTablet: tablet c23f5f085dd84d68beb277676225e7e2 deleted and replaced by tablet a6a75f8dc80a46dcbf17bd838b9a4145
I20250112 20:51:44.195037 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c23f5f085dd84d68beb277676225e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet a6a75f8dc80a46dcbf17bd838b9a4145) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:44.195104 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc90d3722fd14d59b9994655cd30ef2b consensus state has no leader
I20250112 20:51:44.195657 883 tablet_service.cc:1514] Processing DeleteTablet for tablet c23f5f085dd84d68beb277676225e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet a6a75f8dc80a46dcbf17bd838b9a4145) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.195849 1178 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.195972 1178 ts_tablet_manager.cc:1397] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250112 20:51:44.196666 1178 raft_consensus.cc:357] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.196921 1178 raft_consensus.cc:383] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.197010 1178 raft_consensus.cc:738] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.197242 1178 consensus_queue.cc:260] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:44.198953 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c23f5f085dd84d68beb277676225e7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23f5f085dd84d68beb277676225e7e2 already in progress: creating tablet
I20250112 20:51:44.199651 1178 ts_tablet_manager.cc:1428] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:44.199929 1178 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.201526 1178 tablet_bootstrap.cc:654] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.209344 1178 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.209473 1178 ts_tablet_manager.cc:1397] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250112 20:51:44.210126 1178 raft_consensus.cc:357] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.210343 1178 raft_consensus.cc:383] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.210415 1178 raft_consensus.cc:738] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.210637 1178 consensus_queue.cc:260] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.213402 1178 ts_tablet_manager.cc:1428] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:44.213670 1178 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.215299 1178 tablet_bootstrap.cc:654] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.216967 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_3300f1667af244cfb434fd02511942b6 [id=91351703475c45e69c55a3ee47c4bb51] alter complete (version 1)
W20250112 20:51:44.218585 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 55) took 816 ms (client timeout 1000 ms). Trace:
W20250112 20:51:44.218699 446 rpcz_store.cc:269] 0112 20:51:43.402612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.441361 (+ 38749us) service_pool.cc:225] Handling call
0112 20:51:43.457750 (+ 16389us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14f96b8 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:44.218547 (+760797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:43.598160 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:43.598276 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:43.598284 (+ 8us) write_op.cc:435] Acquired schema lock
0112 20:51:43.598288 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:43.598406 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:43.598418 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:43.598422 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 20:51:43.598565 (+ 143us) tablet.cc:698] Row locks acquired
0112 20:51:43.598568 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:43.598605 (+ 37us) write_op.cc:270] Start()
0112 20:51:43.598639 (+ 34us) write_op.cc:276] Timestamp: P: 1736715103598537 usec, L: 0
0112 20:51:43.598643 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:43.660093 (+ 61450us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x616000042ab8. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 000000000096ceda 00007fa2e97cbfa5 00007fa2e98940ab 00007fa2e986fc28 00007fa2e986f4d5 00007fa2eabf7eb5 00007fa2eabf69d9 00007fa2eabfee00 00007fa2eabfec0d 0000000000a71ecc 00007fa2de03cb64
0112 20:51:43.660309 (+ 216us) log.cc:844] Serialized 901 byte log entry
0112 20:51:43.660473 (+ 164us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 000000000096ceda 00007fa2e986f632 00007fa2eabf7eb5 00007fa2eabf69d9 00007fa2eabfee00 00007fa2eabfec0d 0000000000a71ecc 00007fa2de03cb64 00007fa2de04018b 00007fa2de040008 0000000000a71ecc
0112 20:51:43.714975 (+ 54502us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.715069 (+ 94us) write_op.cc:301] APPLY: starting
0112 20:51:43.715088 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:43.715133 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:43.715136 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:43.715218 (+ 82us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:43.715251 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 20:51:43.715257 (+ 6us) write_op.cc:312] APPLY: finished
0112 20:51:43.715341 (+ 84us) log.cc:844] Serialized 29 byte log entry
0112 20:51:43.715608 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.715635 (+ 27us) write_op.cc:454] Released schema lock
0112 20:51:43.715644 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:43.744932 (+ 29288us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x61700006ffd0. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e979a000 00007fa2e97a1c4f 00007fa2e97a193d 0000000000a71ecc 00007fa2de03cb64 00007fa2de04018b 00007fa2de040008 0000000000a71ecc 00007fa2de01b0bd
Related trace 'op':
0112 20:51:43.773305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:43.773402 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:43.773408 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:43.773410 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:43.773495 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:43.773504 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:43.773508 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:43.773557 (+ 49us) tablet.cc:698] Row locks acquired
0112 20:51:43.773559 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:43.773587 (+ 28us) write_op.cc:270] Start()
0112 20:51:43.773616 (+ 29us) write_op.cc:276] Timestamp: P: 1736715103773555 usec, L: 0
0112 20:51:43.773619 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:43.773784 (+ 165us) log.cc:844] Serialized 356 byte log entry
0112 20:51:44.216303 (+442519us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.216385 (+ 82us) write_op.cc:301] APPLY: starting
0112 20:51:44.216400 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:44.216453 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:44.216486 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:44.216534 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:44.216565 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:44.216572 (+ 7us) write_op.cc:312] APPLY: finished
0112 20:51:44.216645 (+ 73us) log.cc:844] Serialized 23 byte log entry
0112 20:51:44.216701 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.216714 (+ 13us) write_op.cc:454] Released schema lock
0112 20:51:44.216719 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":5865,"reported_tablets":30,"spinlock_wait_cycles":34418560,"child_traces":[["op",{"apply.queue_time_us":40,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":645,"num_ops":2,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":1826,"prepare.run_wall_time_us":62381,"raft.queue_time_us":3117,"raft.run_cpu_time_us":1070,"raft.run_wall_time_us":81313,"replication_time_us":116288,"spinlock_wait_cycles":327400448}],["op",{"apply.queue_time_us":34,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":390,"num_ops":1,"prepare.queue_time_us":32827,"prepare.run_cpu_time_us":577,"prepare.run_wall_time_us":11530,"replication_time_us":442531}]]}
I20250112 20:51:44.225805 1163 tablet_bootstrap.cc:654] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.226680 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.226703 884 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.227645 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.227929 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.228134 753 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.228358 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.228538 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.228718 747 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.229156 741 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.229377 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.229528 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.229635 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.229751 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.231128 2521 consensus_queue.cc:1035] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.233438 1185 tablet_replica.cc:331] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.233623 1185 raft_consensus.cc:2238] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.233812 1185 pending_rounds.cc:62] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:44.233855 1185 pending_rounds.cc:69] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585064427667456 op_type: NO_OP noop_request { }
I20250112 20:51:44.233935 1185 raft_consensus.cc:2883] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:44.233995 1185 raft_consensus.cc:2267] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.237346 1186 ts_tablet_manager.cc:1918] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.237699 1186 log.cc:1198] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/903027fe49e34a308108356ba89a9c3a
I20250112 20:51:44.238258 1186 ts_tablet_manager.cc:1939] T 903027fe49e34a308108356ba89a9c3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.238642 1185 ts_tablet_manager.cc:1905] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.240190 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 830) took 1314 ms. Trace:
I20250112 20:51:44.240708 1186 rpcz_store.cc:276] 0112 20:51:42.925366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.925412 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:44.162464 (+1237052us) ts_tablet_manager.cc:1145] Deleting tablet 903027fe49e34a308108356ba89a9c3a
0112 20:51:44.162479 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.211302 (+ 48823us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.232737 (+ 21435us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.240171 (+ 7434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236883}
I20250112 20:51:44.241462 1186 tablet_replica.cc:331] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.241832 1186 raft_consensus.cc:2238] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.242347 1186 raft_consensus.cc:2267] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.243734 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 903027fe49e34a308108356ba89a9c3a (table default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]) successfully deleted
W20250112 20:51:44.247454 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ad1a4b5ac2f94874b99b3ef790a034fd because the tablet was not found. No further retry: Not found: Tablet not found: ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:44.248893 2521 raft_consensus.cc:491] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.249023 2521 raft_consensus.cc:513] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.249594 2521 leader_election.cc:290] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.250427 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.250437 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.250607 776 raft_consensus.cc:2463] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
W20250112 20:51:44.251103 847 leader_election.cc:343] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.251457 847 leader_election.cc:304] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:44.251850 2108 raft_consensus.cc:491] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.251955 2108 raft_consensus.cc:513] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.252444 2108 leader_election.cc:290] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.252565 2108 raft_consensus.cc:2798] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.252619 2108 raft_consensus.cc:491] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.252668 2108 raft_consensus.cc:3054] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.252938 1186 ts_tablet_manager.cc:1905] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.253553 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.254352 1185 ts_tablet_manager.cc:1918] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.254499 1185 log.cc:1198] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9856e6b625d04f17b0869cee08f69142
I20250112 20:51:44.254880 1185 ts_tablet_manager.cc:1939] T 9856e6b625d04f17b0869cee08f69142 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.256011 1163 tablet_bootstrap.cc:492] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.256146 1163 ts_tablet_manager.cc:1397] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.008s
I20250112 20:51:44.256441 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.256839 1163 raft_consensus.cc:357] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.256999 1163 raft_consensus.cc:383] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.257040 1163 raft_consensus.cc:738] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.257221 1163 consensus_queue.cc:260] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.257678 1163 ts_tablet_manager.cc:1428] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:44.257901 1163 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.259392 1163 tablet_bootstrap.cc:654] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.260808 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9856e6b625d04f17b0869cee08f69142 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:44.254299 1178 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.264219 1178 ts_tablet_manager.cc:1397] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.008s
I20250112 20:51:44.264932 1178 raft_consensus.cc:357] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.265100 1178 raft_consensus.cc:383] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.265149 1178 raft_consensus.cc:738] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.265322 1178 consensus_queue.cc:260] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.265760 1178 ts_tablet_manager.cc:1428] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:44.255825 2108 raft_consensus.cc:513] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.256248 847 leader_election.cc:343] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.266413 2108 leader_election.cc:290] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:44.266533 847 leader_election.cc:343] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.266634 847 leader_election.cc:304] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.267094 2415 raft_consensus.cc:2743] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.267158 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bfe92375d9a04e5aa7fc64840b0bc050" } new_table_name: "default.Table_023f7c8ba47d421fbb789a60f7830d0b" modify_external_catalogs: true
I20250112 20:51:44.267696 1178 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.268252 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:44.268529 773 raft_consensus.cc:3054] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.268795 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f669a739d94f99a5b7e9ccccd29545" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.269260 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_968d00519a1942b78b337dc9f4980744" } new_table_name: "default.Table_19d42abaaab84ae99ac8478caf338959" modify_external_catalogs: true
I20250112 20:51:44.269510 1178 tablet_bootstrap.cc:654] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.271279 773 raft_consensus.cc:2463] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:44.272913 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b403df49b35f4c1db5345f2e317fc890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:44.273300 847 leader_election.cc:304] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:44.273527 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b403df49b35f4c1db5345f2e317fc890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:44.273566 847 leader_election.cc:343] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.273938 2648 raft_consensus.cc:2798] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.274055 2648 raft_consensus.cc:695] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:44.274296 2648 consensus_queue.cc:237] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.281529 1186 ts_tablet_manager.cc:1918] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.284889 1186 log.cc:1198] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/54ab0ae10f09469fa4f913e049002481
I20250112 20:51:44.285444 1186 ts_tablet_manager.cc:1939] T 54ab0ae10f09469fa4f913e049002481 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.286235 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 831) took 1360 ms. Trace:
I20250112 20:51:44.288105 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_77867f973c5547568b64d2b3448da1d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.288770 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77867f973c5547568b64d2b3448da1d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.289942 1163 tablet_bootstrap.cc:492] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.290093 1163 ts_tablet_manager.cc:1397] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250112 20:51:44.290824 1163 raft_consensus.cc:357] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.291019 1163 raft_consensus.cc:383] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.291085 1163 raft_consensus.cc:738] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.291283 1163 consensus_queue.cc:260] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.291883 1163 ts_tablet_manager.cc:1428] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.292151 1163 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.292831 1186 rpcz_store.cc:276] 0112 20:51:42.925597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.925678 (+ 81us) service_pool.cc:225] Handling call
0112 20:51:44.241420 (+1315742us) ts_tablet_manager.cc:1145] Deleting tablet 54ab0ae10f09469fa4f913e049002481
0112 20:51:44.241429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.266896 (+ 25467us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.281438 (+ 14542us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.286218 (+ 4780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315615}
I20250112 20:51:44.293381 1186 tablet_replica.cc:331] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.293638 1186 raft_consensus.cc:2238] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.293726 1163 tablet_bootstrap.cc:654] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.293876 1186 raft_consensus.cc:2267] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.294137 1178 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.294274 1178 ts_tablet_manager.cc:1397] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20250112 20:51:44.294492 1186 ts_tablet_manager.cc:1905] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.294909 1178 raft_consensus.cc:357] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.295075 1178 raft_consensus.cc:383] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.295130 1178 raft_consensus.cc:738] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.295153 2208 raft_consensus.cc:491] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.295310 2208 raft_consensus.cc:513] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.295327 1178 consensus_queue.cc:260] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.295972 2208 leader_election.cc:290] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.296584 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.296702 1178 ts_tablet_manager.cc:1428] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.296747 912 raft_consensus.cc:2463] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.297163 710 leader_election.cc:304] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.297454 2208 raft_consensus.cc:2798] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.297531 2208 raft_consensus.cc:491] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.297590 2208 raft_consensus.cc:3054] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.298372 1178 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.299608 1178 tablet_bootstrap.cc:654] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.299952 2208 raft_consensus.cc:513] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.301087 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.301259 912 raft_consensus.cc:3054] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.303671 912 raft_consensus.cc:2463] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.304100 710 leader_election.cc:304] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.304373 2632 raft_consensus.cc:2798] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.304488 2632 raft_consensus.cc:695] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.304733 2632 consensus_queue.cc:237] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.305920 2208 leader_election.cc:290] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.307358 1186 ts_tablet_manager.cc:1918] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.307595 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f1cd4dd5cbfb498b9337d35cac1bd249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.308374 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1cd4dd5cbfb498b9337d35cac1bd249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.310357 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 54ab0ae10f09469fa4f913e049002481 (table default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]) successfully deleted
I20250112 20:51:44.314337 1186 log.cc:1198] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d706f88f14ab4fd9a459944863afaacd
I20250112 20:51:44.314792 1186 ts_tablet_manager.cc:1939] T d706f88f14ab4fd9a459944863afaacd P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.315805 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 832) took 1390 ms. Trace:
I20250112 20:51:44.315966 1186 rpcz_store.cc:276] 0112 20:51:42.925819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.925892 (+ 73us) service_pool.cc:225] Handling call
0112 20:51:44.293321 (+1367429us) ts_tablet_manager.cc:1145] Deleting tablet d706f88f14ab4fd9a459944863afaacd
0112 20:51:44.293331 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.301670 (+ 8339us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.307274 (+ 5604us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.315784 (+ 8510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367302}
W20250112 20:51:44.319429 710 consensus_peers.cc:487] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d706f88f14ab4fd9a459944863afaacd. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.319932 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d706f88f14ab4fd9a459944863afaacd (table default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]) successfully deleted
I20250112 20:51:44.320350 1186 tablet_replica.cc:331] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.320578 1186 raft_consensus.cc:2238] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.320907 1186 raft_consensus.cc:2267] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.329739 1186 ts_tablet_manager.cc:1905] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.329787 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b04ae76622ac46f5a584a582e372bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ae76622ac46f5a584a582e372bc05" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35199 } } peers { permanent_uuid: "b985fd656ce44131bc91ccfeb79587c5" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39271 } } peers { permanent_uuid: "0f5c09e899184de6b770dca873501062" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 45153 } } }
W20250112 20:51:44.332214 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 161951bfabff485dbf7dceeac2d4128d consensus state has no leader
I20250112 20:51:44.333936 2108 raft_consensus.cc:491] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.334043 2108 raft_consensus.cc:513] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.334117 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.334546 2108 leader_election.cc:290] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.335302 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.335855 847 leader_election.cc:343] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.335983 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:44.336447 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:44.336987 847 leader_election.cc:343] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.337090 847 leader_election.cc:304] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.337474 2108 raft_consensus.cc:2743] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.337734 1185 tablet_replica.cc:331] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.337996 1185 raft_consensus.cc:2238] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.338261 1185 pending_rounds.cc:62] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:44.338316 1185 pending_rounds.cc:69] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585064630317056 op_type: NO_OP noop_request { }
I20250112 20:51:44.338397 1178 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.338459 1185 raft_consensus.cc:2883] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:44.338516 1178 ts_tablet_manager.cc:1397] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.000s
I20250112 20:51:44.338554 1185 raft_consensus.cc:2267] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.339182 1178 raft_consensus.cc:357] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.339387 1178 raft_consensus.cc:383] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.339447 1178 raft_consensus.cc:738] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.339643 1178 consensus_queue.cc:260] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.340112 1178 ts_tablet_manager.cc:1428] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:44.340376 1178 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.342633 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.342712 2563 raft_consensus.cc:491] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.342839 2563 raft_consensus.cc:513] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.343063 1178 tablet_bootstrap.cc:654] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.343501 2563 leader_election.cc:290] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.344481 1042 raft_consensus.cc:3054] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.345852 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.346887 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11a892641554e4db9bdd558068cac40" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.347450 983 leader_election.cc:304] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.347939 2636 raft_consensus.cc:2743] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.349159 2208 consensus_queue.cc:1035] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.356164 1185 ts_tablet_manager.cc:1905] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.357020 2613 tablet.cc:1721] T 0dc920e92de545fc961f82e7d0de28cd P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.360117 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet c23f5f085dd84d68beb277676225e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet a6a75f8dc80a46dcbf17bd838b9a4145) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.360296 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b788acf8047acb34ca2596bbf141b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.361286 1178 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.361480 1178 ts_tablet_manager.cc:1397] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20250112 20:51:44.362224 1178 raft_consensus.cc:357] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.362450 1178 raft_consensus.cc:383] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.362529 1178 raft_consensus.cc:738] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.362747 1178 consensus_queue.cc:260] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.363916 884 tablet_service.cc:1467] Processing CreateTablet for tablet c6bdfecd5d9142d5b008c613864c8217 (DEFAULT_TABLE table=default.Table_527edcc30df04186a34eb7b085689535 [id=304ee27803dd4d6e9613f8b9ae42e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.364288 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bdfecd5d9142d5b008c613864c8217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.364298 1178 ts_tablet_manager.cc:1428] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 20:51:44.365329 1178 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.365931 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:44.366220 447 catalog_manager.cc:5526] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:44.366969 1178 tablet_bootstrap.cc:654] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.370177 881 tablet_service.cc:1467] Processing CreateTablet for tablet a15b6492862048fe8cda607b80a49490 (DEFAULT_TABLE table=default.Table_72ecf6832e6c42458c48a0ffe1a66ad4 [id=fc9859a2d46242bdb77be66a3d30cf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.370329 892 tablet_service.cc:1467] Processing CreateTablet for tablet f7d10a9f07cf426eb818783bac2d1833 (DEFAULT_TABLE table=default.Table_14e5a2e03fd74b87bf2afbb8a1688f30 [id=d8733c874d664753a058b980402073b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.370329 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9705feb15f384b5185935eacc521d0e0 (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.370559 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a15b6492862048fe8cda607b80a49490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.370779 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d10a9f07cf426eb818783bac2d1833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.370939 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9705feb15f384b5185935eacc521d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.370177 875 tablet_service.cc:1467] Processing CreateTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e (DEFAULT_TABLE table=default.Table_48039b63c7ee45558ce7a911e805f60a [id=0cacf5873a7249e596746f4ffe72776b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.371208 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fa87d8f0c64f7aa616882fe399ef0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:44.371316 710 leader_election.cc:343] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:44.371809 710 leader_election.cc:343] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.373585 1029 tablet_service.cc:1467] Processing CreateTablet for tablet c6bdfecd5d9142d5b008c613864c8217 (DEFAULT_TABLE table=default.Table_527edcc30df04186a34eb7b085689535 [id=304ee27803dd4d6e9613f8b9ae42e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.373745 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e (DEFAULT_TABLE table=default.Table_48039b63c7ee45558ce7a911e805f60a [id=0cacf5873a7249e596746f4ffe72776b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.373965 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bdfecd5d9142d5b008c613864c8217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.374158 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fa87d8f0c64f7aa616882fe399ef0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.374413 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9705feb15f384b5185935eacc521d0e0 (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.374743 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9705feb15f384b5185935eacc521d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.377209 745 tablet_service.cc:1467] Processing CreateTablet for tablet c6bdfecd5d9142d5b008c613864c8217 (DEFAULT_TABLE table=default.Table_527edcc30df04186a34eb7b085689535 [id=304ee27803dd4d6e9613f8b9ae42e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.377372 754 tablet_service.cc:1467] Processing CreateTablet for tablet a15b6492862048fe8cda607b80a49490 (DEFAULT_TABLE table=default.Table_72ecf6832e6c42458c48a0ffe1a66ad4 [id=fc9859a2d46242bdb77be66a3d30cf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.377609 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bdfecd5d9142d5b008c613864c8217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.377833 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a15b6492862048fe8cda607b80a49490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.380445 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9705feb15f384b5185935eacc521d0e0 (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.380766 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9705feb15f384b5185935eacc521d0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.381036 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a15b6492862048fe8cda607b80a49490 (DEFAULT_TABLE table=default.Table_72ecf6832e6c42458c48a0ffe1a66ad4 [id=fc9859a2d46242bdb77be66a3d30cf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.381364 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a15b6492862048fe8cda607b80a49490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.384065 737 tablet_service.cc:1467] Processing CreateTablet for tablet f7d10a9f07cf426eb818783bac2d1833 (DEFAULT_TABLE table=default.Table_14e5a2e03fd74b87bf2afbb8a1688f30 [id=d8733c874d664753a058b980402073b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.384361 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d10a9f07cf426eb818783bac2d1833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.386940 754 tablet_service.cc:1467] Processing CreateTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e (DEFAULT_TABLE table=default.Table_48039b63c7ee45558ce7a911e805f60a [id=0cacf5873a7249e596746f4ffe72776b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.387236 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fa87d8f0c64f7aa616882fe399ef0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.401242 1164 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.401388 1164 ts_tablet_manager.cc:1397] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.299s user 0.019s sys 0.028s
I20250112 20:51:44.402990 1164 raft_consensus.cc:357] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.403184 1164 raft_consensus.cc:383] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.403239 1164 raft_consensus.cc:738] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.403435 1164 consensus_queue.cc:260] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.406397 892 tablet_service.cc:1467] Processing CreateTablet for tablet f6e0049fbf174962931c4c3d713535e1 (DEFAULT_TABLE table=default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465 [id=3664f37c48304fe59a9f5ee97eb83de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.406571 875 tablet_service.cc:1467] Processing CreateTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 (DEFAULT_TABLE table=default.Table_e132d9ac027d44bc8aa20fbdacd57a62 [id=a39d96d4df4f46758ec86110834368c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.406780 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e0049fbf174962931c4c3d713535e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.407029 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbdbf416f64011b431f5188e0a06c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.407647 737 tablet_service.cc:1467] Processing CreateTablet for tablet f6e0049fbf174962931c4c3d713535e1 (DEFAULT_TABLE table=default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465 [id=3664f37c48304fe59a9f5ee97eb83de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.407924 741 tablet_service.cc:1467] Processing CreateTablet for tablet 6b440da020d947dda2553d65dd6a4bdf (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.408075 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e0049fbf174962931c4c3d713535e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.408238 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b440da020d947dda2553d65dd6a4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.408492 756 tablet_service.cc:1467] Processing CreateTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 (DEFAULT_TABLE table=default.Table_e132d9ac027d44bc8aa20fbdacd57a62 [id=a39d96d4df4f46758ec86110834368c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.408762 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbdbf416f64011b431f5188e0a06c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.409463 884 tablet_service.cc:1467] Processing CreateTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 (DEFAULT_TABLE table=default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3 [id=dcd5981e8d9345f58fe9f53eb64ca681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.409797 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.409040 745 tablet_service.cc:1467] Processing CreateTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 (DEFAULT_TABLE table=default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3 [id=dcd5981e8d9345f58fe9f53eb64ca681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.410284 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.412940 881 tablet_service.cc:1467] Processing CreateTablet for tablet 6b440da020d947dda2553d65dd6a4bdf (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.413254 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b440da020d947dda2553d65dd6a4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.414134 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f7d10a9f07cf426eb818783bac2d1833 (DEFAULT_TABLE table=default.Table_14e5a2e03fd74b87bf2afbb8a1688f30 [id=d8733c874d664753a058b980402073b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.414510 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d10a9f07cf426eb818783bac2d1833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.415535 1014 tablet_service.cc:1467] Processing CreateTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 (DEFAULT_TABLE table=default.Table_e132d9ac027d44bc8aa20fbdacd57a62 [id=a39d96d4df4f46758ec86110834368c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.415835 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbdbf416f64011b431f5188e0a06c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.416620 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6b440da020d947dda2553d65dd6a4bdf (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.416937 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b440da020d947dda2553d65dd6a4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.420130 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f6e0049fbf174962931c4c3d713535e1 (DEFAULT_TABLE table=default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465 [id=3664f37c48304fe59a9f5ee97eb83de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.420447 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6e0049fbf174962931c4c3d713535e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:44.424738 710 consensus_peers.cc:487] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.425093 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 (DEFAULT_TABLE table=default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3 [id=dcd5981e8d9345f58fe9f53eb64ca681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.425429 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.425628 1186 ts_tablet_manager.cc:1918] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.425720 1186 log.cc:1198] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:44.426051 1186 ts_tablet_manager.cc:1939] T 2ab635fec8a64efd8e0c527d913ccd7d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.426330 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 833) took 1500 ms. Trace:
I20250112 20:51:44.426417 1186 rpcz_store.cc:276] 0112 20:51:42.926296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.926336 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:44.320309 (+1393973us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:44.320318 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.368306 (+ 47988us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.425551 (+ 57245us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.426315 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1393843}
I20250112 20:51:44.426720 1186 tablet_replica.cc:331] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.426826 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 2ab635fec8a64efd8e0c527d913ccd7d (table default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]) successfully deleted
I20250112 20:51:44.426929 1186 raft_consensus.cc:2238] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.427088 1186 raft_consensus.cc:2267] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.427551 1186 ts_tablet_manager.cc:1905] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.431988 1042 raft_consensus.cc:3054] T 64f1119a719b4c0fa47650dc1fb1c27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.433380 1164 ts_tablet_manager.cc:1428] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250112 20:51:44.434506 2447 ts_tablet_manager.cc:1918] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.434603 2447 log.cc:1198] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/944b71897bcf4304bd9f7bb7adef34e5
I20250112 20:51:44.436501 1185 ts_tablet_manager.cc:1918] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.437235 2447 ts_tablet_manager.cc:1939] T 944b71897bcf4304bd9f7bb7adef34e5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.439780 2447 tablet_replica.cc:331] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.440176 881 tablet_service.cc:1467] Processing CreateTablet for tablet 70f46394c9cb4947bc0346a03796ead6 (DEFAULT_TABLE table=default.Table_2be21f44001347a1ac95041d68da6a79 [id=0dfdd3e2880240668908863273aeebaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.436877 1164 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.442847 1164 tablet_bootstrap.cc:654] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.440840 2447 raft_consensus.cc:2238] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.443048 1185 log.cc:1198] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:44.443701 2447 raft_consensus.cc:2267] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.436873 2632 raft_consensus.cc:491] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.444193 2632 raft_consensus.cc:513] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.445487 2632 leader_election.cc:290] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.445585 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.445766 912 raft_consensus.cc:2463] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.446148 710 leader_election.cc:304] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.446352 1185 ts_tablet_manager.cc:1939] T efa7f6a1e9804e6ab822f9d4c004b1b3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.447863 2447 ts_tablet_manager.cc:1905] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.450853 875 tablet_service.cc:1467] Processing CreateTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 (DEFAULT_TABLE table=default.Table_65cffe4814de46449661f915c2371007 [id=0caeede6d29c4c8cae479bd5f076e764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.452877 2632 raft_consensus.cc:2798] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.453030 2632 raft_consensus.cc:491] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.453178 2632 raft_consensus.cc:3054] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.453601 1185 tablet_replica.cc:331] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.455222 1185 raft_consensus.cc:2238] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.456270 2632 raft_consensus.cc:513] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.456565 892 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d5d182827465abd3799d3ad9a6484 (DEFAULT_TABLE table=default.Table_2e8ea09f0c2c4ccea9540106b863d8a8 [id=e33fce0568264053ab59102945009b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.458221 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.458396 912 raft_consensus.cc:3054] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:44.459542 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90487cfd33164562855831e20b41ce8e consensus state has no leader
I20250112 20:51:44.460949 912 raft_consensus.cc:2463] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.458667 2632 leader_election.cc:290] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.461382 710 leader_election.cc:304] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.457528 1185 pending_rounds.cc:62] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:44.463002 2585 raft_consensus.cc:2798] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.463140 2585 raft_consensus.cc:695] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.463255 1163 tablet_bootstrap.cc:492] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.463380 1163 ts_tablet_manager.cc:1397] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.171s user 0.015s sys 0.000s
I20250112 20:51:44.462818 1185 pending_rounds.cc:69] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585063977041920 op_type: NO_OP noop_request { }
I20250112 20:51:44.463390 2585 consensus_queue.cc:237] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.463519 1185 raft_consensus.cc:2883] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:44.464134 1163 raft_consensus.cc:357] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.464309 1163 raft_consensus.cc:383] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.464355 1163 raft_consensus.cc:738] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.464140 1185 raft_consensus.cc:2267] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.464548 1163 consensus_queue.cc:260] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.465088 1163 ts_tablet_manager.cc:1428] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:44.465342 1163 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.466977 1163 tablet_bootstrap.cc:654] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.468430 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b4a1d95f60f74be2a1f424c514f6efc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.469033 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4a1d95f60f74be2a1f424c514f6efc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.469475 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_13c00ee8163d4ea0b5bd8ed71ff8a80c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.469980 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13c00ee8163d4ea0b5bd8ed71ff8a80c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.471331 1185 ts_tablet_manager.cc:1905] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.479411 884 tablet_service.cc:1467] Processing CreateTablet for tablet 90487cfd33164562855831e20b41ce8e (DEFAULT_TABLE table=default.Table_968d00519a1942b78b337dc9f4980744 [id=3e52958a6a074922b3f8e95cdaef8d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.480932 747 tablet_service.cc:1467] Processing CreateTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 (DEFAULT_TABLE table=default.Table_65cffe4814de46449661f915c2371007 [id=0caeede6d29c4c8cae479bd5f076e764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.481344 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.482924 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (table default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]) successfully deleted
I20250112 20:51:44.484995 753 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d5d182827465abd3799d3ad9a6484 (DEFAULT_TABLE table=default.Table_2e8ea09f0c2c4ccea9540106b863d8a8 [id=e33fce0568264053ab59102945009b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.485030 741 tablet_service.cc:1467] Processing CreateTablet for tablet 90487cfd33164562855831e20b41ce8e (DEFAULT_TABLE table=default.Table_968d00519a1942b78b337dc9f4980744 [id=3e52958a6a074922b3f8e95cdaef8d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.485345 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5d182827465abd3799d3ad9a6484. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.485538 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90487cfd33164562855831e20b41ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.492192 754 tablet_service.cc:1467] Processing CreateTablet for tablet 70f46394c9cb4947bc0346a03796ead6 (DEFAULT_TABLE table=default.Table_2be21f44001347a1ac95041d68da6a79 [id=0dfdd3e2880240668908863273aeebaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.494894 2108 consensus_queue.cc:1035] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.506079 2108 raft_consensus.cc:491] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.506254 2108 raft_consensus.cc:513] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.506955 2108 leader_election.cc:290] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.507898 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.508582 847 leader_election.cc:343] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.514825 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1d050efe434c480199f7df0832274841" } modify_external_catalogs: true
I20250112 20:51:44.515169 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1d050efe434c480199f7df0832274841" } modify_external_catalogs: true
I20250112 20:51:44.517374 2447 ts_tablet_manager.cc:1918] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.517927 2447 log.cc:1198] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/64887b3578284cb286ee122ab8c626c4
I20250112 20:51:44.493126 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f46394c9cb4947bc0346a03796ead6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.509125 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.511660 1185 ts_tablet_manager.cc:1918] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.509114 2208 raft_consensus.cc:491] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.520007 2447 ts_tablet_manager.cc:1939] T 64887b3578284cb286ee122ab8c626c4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.519941 1185 log.cc:1198] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d706f88f14ab4fd9a459944863afaacd
W20250112 20:51:44.520378 847 leader_election.cc:343] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.520514 847 leader_election.cc:304] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.520417 2208 raft_consensus.cc:513] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.520888 1185 ts_tablet_manager.cc:1939] T d706f88f14ab4fd9a459944863afaacd P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.520985 2108 raft_consensus.cc:2743] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.522468 2208 leader_election.cc:290] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.522689 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d706f88f14ab4fd9a459944863afaacd (table default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]) successfully deleted
I20250112 20:51:44.523350 1185 tablet_replica.cc:331] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.523519 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 944b71897bcf4304bd9f7bb7adef34e5 (table default.Table_15613426e60a4f298722927f651c546d [id=0b1b46fd11df408191be31b40914a6a8]) successfully deleted
I20250112 20:51:44.523708 1185 raft_consensus.cc:2238] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.523895 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.524053 1185 raft_consensus.cc:2267] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.524219 909 raft_consensus.cc:2463] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.524622 1163 tablet_bootstrap.cc:492] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.524760 1163 ts_tablet_manager.cc:1397] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20250112 20:51:44.525027 710 leader_election.cc:304] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.525411 2208 raft_consensus.cc:2798] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.525571 2208 raft_consensus.cc:491] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.525593 1185 ts_tablet_manager.cc:1905] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.525628 1163 raft_consensus.cc:357] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.525835 1163 raft_consensus.cc:383] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.525710 2208 raft_consensus.cc:3054] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.526104 1163 raft_consensus.cc:738] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.526355 1163 consensus_queue.cc:260] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.526921 1163 ts_tablet_manager.cc:1428] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:44.527194 1163 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.528723 1163 tablet_bootstrap.cc:654] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.529752 2208 raft_consensus.cc:513] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.534008 1178 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.534891 1178 ts_tablet_manager.cc:1397] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.170s user 0.010s sys 0.032s
I20250112 20:51:44.535856 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b" } modify_external_catalogs: true
I20250112 20:51:44.536064 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_83f5c84fabfb4dcda846ea36fa6d3a9b" } modify_external_catalogs: true
I20250112 20:51:44.530337 1164 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.537007 1164 ts_tablet_manager.cc:1397] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.010s
I20250112 20:51:44.538275 1164 raft_consensus.cc:357] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.538533 1164 raft_consensus.cc:383] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.538614 1164 raft_consensus.cc:738] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.538856 1164 consensus_queue.cc:260] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.539023 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 64887b3578284cb286ee122ab8c626c4 (table default.Table_144f2be878e14cb2901b6993ebee9191 [id=7724b3c7ec9a4e4982dfe1e9136fd36d]) successfully deleted
W20250112 20:51:44.539392 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1748fb558c4d48f49432840055d71c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1748fb558c4d48f49432840055d71c67 already in progress: creating tablet
W20250112 20:51:44.539690 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet bf66492d01a64eb1b842acc279b8ee09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf66492d01a64eb1b842acc279b8ee09 already in progress: creating tablet
W20250112 20:51:44.540009 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 1fcaaa7af46c48fbb69589e6013968c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fcaaa7af46c48fbb69589e6013968c0 already in progress: creating tablet
W20250112 20:51:44.540251 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e0fd4177f9394a5a82d4f5e02369b3b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fd4177f9394a5a82d4f5e02369b3b6 already in progress: creating tablet
I20250112 20:51:44.540261 1164 ts_tablet_manager.cc:1428] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:44.540570 1164 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.542407 1164 tablet_bootstrap.cc:654] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.547343 2208 leader_election.cc:290] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.547452 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.547719 909 raft_consensus.cc:3054] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.548169 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5d182827465abd3799d3ad9a6484. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.548391 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f46394c9cb4947bc0346a03796ead6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.549407 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90487cfd33164562855831e20b41ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.552876 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.552932 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c45ac747ab7f4990a03a39553424e951
I20250112 20:51:44.553421 455 catalog_manager.cc:6596] ReplaceTablet: tablet c45ac747ab7f4990a03a39553424e951 deleted and replaced by tablet c950294baf3046a09d2202912ab8c8c8
I20250112 20:51:44.555828 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7ca72a25393a40d9bdbbb1d078b7cff2" } modify_external_catalogs: true
I20250112 20:51:44.556017 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7ca72a25393a40d9bdbbb1d078b7cff2" } modify_external_catalogs: true
I20250112 20:51:44.556361 909 raft_consensus.cc:2463] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.556701 710 leader_election.cc:304] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.557345 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ac747ab7f4990a03a39553424e951 with delete_type TABLET_DATA_DELETED (replaced by tablet c950294baf3046a09d2202912ab8c8c8) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.557551 2208 raft_consensus.cc:2798] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.557745 2208 raft_consensus.cc:695] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.557973 2208 consensus_queue.cc:237] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.530486 2447 tablet_replica.cc:331] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.531421 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 90487cfd33164562855831e20b41ce8e (DEFAULT_TABLE table=default.Table_968d00519a1942b78b337dc9f4980744 [id=3e52958a6a074922b3f8e95cdaef8d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.559671 2447 raft_consensus.cc:2238] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.531579 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 70f46394c9cb4947bc0346a03796ead6 (DEFAULT_TABLE table=default.Table_2be21f44001347a1ac95041d68da6a79 [id=0dfdd3e2880240668908863273aeebaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.559855 2447 raft_consensus.cc:2267] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.532197 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 (DEFAULT_TABLE table=default.Table_65cffe4814de46449661f915c2371007 [id=0caeede6d29c4c8cae479bd5f076e764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.532486 1020 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d5d182827465abd3799d3ad9a6484 (DEFAULT_TABLE table=default.Table_2e8ea09f0c2c4ccea9540106b863d8a8 [id=e33fce0568264053ab59102945009b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.559834 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90487cfd33164562855831e20b41ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.532670 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.532874 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6b4882a3d744bda5677aa9e01a53f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.560434 2447 ts_tablet_manager.cc:1905] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.533404 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.539620 2563 raft_consensus.cc:491] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.560767 2563 raft_consensus.cc:513] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.561350 2563 leader_election.cc:290] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.561915 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5d182827465abd3799d3ad9a6484. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.564914 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.565274 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6092a31de9bc47e4bae5f5aebb180c6d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.567723 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f46394c9cb4947bc0346a03796ead6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:44.569520 983 leader_election.cc:343] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:44.569628 983 leader_election.cc:304] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.570058 2521 raft_consensus.cc:491] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.570197 2521 raft_consensus.cc:513] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.570739 2521 leader_election.cc:290] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:44.571391 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29316b9d2441bca97f56bdb0e56f0d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.571885 847 leader_election.cc:343] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.572103 2563 raft_consensus.cc:2743] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.572542 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29316b9d2441bca97f56bdb0e56f0d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.572713 776 raft_consensus.cc:2463] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:44.573156 847 leader_election.cc:304] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:44.573542 2521 raft_consensus.cc:2798] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.573611 2521 raft_consensus.cc:491] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.573658 2521 raft_consensus.cc:3054] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.574311 1164 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.574419 1164 ts_tablet_manager.cc:1397] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250112 20:51:44.575129 1164 raft_consensus.cc:357] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.575304 1164 raft_consensus.cc:383] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.575345 1164 raft_consensus.cc:738] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.575520 1164 consensus_queue.cc:260] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.575866 2521 raft_consensus.cc:513] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.576349 2521 leader_election.cc:290] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:44.576351 1164 ts_tablet_manager.cc:1428] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.576627 1164 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.576902 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ac747ab7f4990a03a39553424e951 with delete_type TABLET_DATA_DELETED (replaced by tablet c950294baf3046a09d2202912ab8c8c8) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.577165 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29316b9d2441bca97f56bdb0e56f0d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.577828 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29316b9d2441bca97f56bdb0e56f0d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:44.578009 776 raft_consensus.cc:3054] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.579583 1164 tablet_bootstrap.cc:654] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.580291 776 raft_consensus.cc:2463] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:44.581045 847 leader_election.cc:304] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:44.581298 847 leader_election.cc:343] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.581640 2108 raft_consensus.cc:491] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.581743 2108 raft_consensus.cc:513] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.582280 2108 leader_election.cc:290] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.582391 2108 raft_consensus.cc:2798] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.582476 2108 raft_consensus.cc:695] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:44.582664 2108 consensus_queue.cc:237] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.584591 1186 ts_tablet_manager.cc:1918] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:44.584679 1186 log.cc:1198] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ba86d0ef63d94a55adf79bb136ea89e8
I20250112 20:51:44.585990 2648 raft_consensus.cc:491] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.586198 2648 raft_consensus.cc:513] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.586400 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.586835 2648 leader_election.cc:290] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:44.586997 847 leader_election.cc:343] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.587572 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.587641 1178 raft_consensus.cc:357] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.588336 2447 ts_tablet_manager.cc:1918] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:44.588202 847 leader_election.cc:343] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.588442 2447 log.cc:1198] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4bd46a422093495ba8b5b03039874a32
I20250112 20:51:44.588393 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.588587 1178 raft_consensus.cc:383] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.588696 1178 raft_consensus.cc:738] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.588737 2447 ts_tablet_manager.cc:1939] T 4bd46a422093495ba8b5b03039874a32 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:44.588872 847 leader_election.cc:343] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.588975 847 leader_election.cc:304] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.589157 2648 raft_consensus.cc:2743] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.589349 1178 consensus_queue.cc:260] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.591918 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.592300 1185 ts_tablet_manager.cc:1918] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.592393 1185 log.cc:1198] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/54ab0ae10f09469fa4f913e049002481
I20250112 20:51:44.592756 1185 ts_tablet_manager.cc:1939] T 54ab0ae10f09469fa4f913e049002481 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:44.592917 847 leader_election.cc:343] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.593230 847 leader_election.cc:304] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.593470 1178 ts_tablet_manager.cc:1428] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.058s user 0.000s sys 0.002s
I20250112 20:51:44.593725 1178 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.595217 2521 raft_consensus.cc:2743] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.597105 1178 tablet_bootstrap.cc:654] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.597266 1186 ts_tablet_manager.cc:1939] T ba86d0ef63d94a55adf79bb136ea89e8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.598549 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 835) took 1245 ms. Trace:
I20250112 20:51:44.598666 1186 rpcz_store.cc:276] 0112 20:51:43.352615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.353202 (+ 587us) service_pool.cc:225] Handling call
0112 20:51:44.426677 (+1073475us) ts_tablet_manager.cc:1145] Deleting tablet ba86d0ef63d94a55adf79bb136ea89e8
0112 20:51:44.426687 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.581932 (+155245us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.584513 (+ 2581us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.598534 (+ 14021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054585}
I20250112 20:51:44.599268 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ba86d0ef63d94a55adf79bb136ea89e8 (table default.Table_fe56f26bf2934585834bfaf76f5a1d08 [id=e0dcbc51ca5c4453a6ef45cf04fdd4e4]) successfully deleted
I20250112 20:51:44.599603 2447 tablet_replica.cc:331] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.599792 2447 raft_consensus.cc:2238] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.599932 2447 raft_consensus.cc:2267] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:44.600319 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet bf66492d01a64eb1b842acc279b8ee09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf66492d01a64eb1b842acc279b8ee09 already in progress: creating tablet
I20250112 20:51:44.600399 2447 ts_tablet_manager.cc:1905] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:44.601073 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e0fd4177f9394a5a82d4f5e02369b3b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fd4177f9394a5a82d4f5e02369b3b6 already in progress: creating tablet
W20250112 20:51:44.602160 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1fcaaa7af46c48fbb69589e6013968c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fcaaa7af46c48fbb69589e6013968c0 already in progress: creating tablet
I20250112 20:51:44.602613 1186 tablet_replica.cc:331] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.603296 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 54ab0ae10f09469fa4f913e049002481 (table default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]) successfully deleted
I20250112 20:51:44.603688 1186 raft_consensus.cc:2238] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.603889 1186 raft_consensus.cc:2267] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:44.604398 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet aa9219fc971c4d15b0d383b20b61e540 because the tablet was not found. No further retry: Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:44.605067 1186 ts_tablet_manager.cc:1905] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:44.605621 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet bc05d287d81c46558acdd9a42edda6f8 because the tablet was not found. No further retry: Not found: Tablet not found: bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:44.605918 1185 tablet_replica.cc:331] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.606128 1185 raft_consensus.cc:2238] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.607146 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9498a04f20dd4cf4a283f17d37ca49ec
I20250112 20:51:44.607635 451 catalog_manager.cc:6596] ReplaceTablet: tablet 9498a04f20dd4cf4a283f17d37ca49ec deleted and replaced by tablet efa3af02669a47bf8b112c68bb1805d8
I20250112 20:51:44.608139 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec with delete_type TABLET_DATA_DELETED (replaced by tablet efa3af02669a47bf8b112c68bb1805d8) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.607394 1185 raft_consensus.cc:2267] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.607968 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec with delete_type TABLET_DATA_DELETED (replaced by tablet efa3af02669a47bf8b112c68bb1805d8) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.609269 1185 ts_tablet_manager.cc:1905] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.612085 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d1817a082b7f4497b916c47271738874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.612638 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1817a082b7f4497b916c47271738874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.613706 1178 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.613837 1178 ts_tablet_manager.cc:1397] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250112 20:51:44.614519 1178 raft_consensus.cc:357] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.614734 1178 raft_consensus.cc:383] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.614817 1178 raft_consensus.cc:738] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:44.615088 710 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:44.615754 710 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:44.616073 710 leader_election.cc:343] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.616668 1178 consensus_queue.cc:260] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.617439 1164 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.617583 1164 ts_tablet_manager.cc:1397] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.001s
I20250112 20:51:44.618497 1178 ts_tablet_manager.cc:1428] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 20:51:44.618497 1164 raft_consensus.cc:357] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.618702 1164 raft_consensus.cc:383] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.618762 1164 raft_consensus.cc:738] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.618939 1164 consensus_queue.cc:260] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.619423 1164 ts_tablet_manager.cc:1428] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:44.619832 1164 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.621160 1178 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.621465 1164 tablet_bootstrap.cc:654] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.622738 1178 tablet_bootstrap.cc:654] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.624320 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d6a0679bf1ed4bbc85f14f62dfcc114d
I20250112 20:51:44.624877 453 catalog_manager.cc:6596] ReplaceTablet: tablet d6a0679bf1ed4bbc85f14f62dfcc114d deleted and replaced by tablet 3ea9425f3e7b46288c81dc8cdade99b2
I20250112 20:51:44.626405 2447 ts_tablet_manager.cc:1918] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.626523 2447 log.cc:1198] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d0829e9dd5854961b5402b5710624612
I20250112 20:51:44.626827 2447 ts_tablet_manager.cc:1939] T d0829e9dd5854961b5402b5710624612 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.627127 884 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d with delete_type TABLET_DATA_DELETED (replaced by tablet 3ea9425f3e7b46288c81dc8cdade99b2) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.627696 2447 tablet_replica.cc:331] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.627724 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3886e5ca0a0b4c71ac1ade9fcf9d8ab2" } modify_external_catalogs: true
I20250112 20:51:44.627933 2447 raft_consensus.cc:2238] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.627982 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3886e5ca0a0b4c71ac1ade9fcf9d8ab2" } modify_external_catalogs: true
I20250112 20:51:44.628082 2447 raft_consensus.cc:2267] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.628590 2447 ts_tablet_manager.cc:1905] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.629058 737 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d with delete_type TABLET_DATA_DELETED (replaced by tablet 3ea9425f3e7b46288c81dc8cdade99b2) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.630626 1178 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.630631 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4bd46a422093495ba8b5b03039874a32 (table default.Table_ff1653ec182d4a2bbdd3c2a01b91d268 [id=1423668dbf434332be3757f520bdf4e8]) successfully deleted
I20250112 20:51:44.630788 1178 ts_tablet_manager.cc:1397] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20250112 20:51:44.631181 2563 raft_consensus.cc:491] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.631289 2563 raft_consensus.cc:513] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.631469 1178 raft_consensus.cc:357] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.631615 1178 raft_consensus.cc:383] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.631657 1178 raft_consensus.cc:738] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.631809 2563 leader_election.cc:290] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.631881 1178 consensus_queue.cc:260] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:44.632962 983 leader_election.cc:343] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: bc05d287d81c46558acdd9a42edda6f8
W20250112 20:51:44.633486 983 leader_election.cc:343] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:44.633605 983 leader_election.cc:304] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.633731 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet c45ac747ab7f4990a03a39553424e951 with delete_type TABLET_DATA_DELETED (replaced by tablet c950294baf3046a09d2202912ab8c8c8) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.633986 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 9498a04f20dd4cf4a283f17d37ca49ec with delete_type TABLET_DATA_DELETED (replaced by tablet efa3af02669a47bf8b112c68bb1805d8) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.634155 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a0679bf1ed4bbc85f14f62dfcc114d with delete_type TABLET_DATA_DELETED (replaced by tablet 3ea9425f3e7b46288c81dc8cdade99b2) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.634384 2563 raft_consensus.cc:2743] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.634953 1178 ts_tablet_manager.cc:1428] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:44.635203 1163 tablet_bootstrap.cc:492] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.635346 1163 ts_tablet_manager.cc:1397] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.009s
I20250112 20:51:44.636157 1178 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.637235 1163 raft_consensus.cc:357] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.637420 1163 raft_consensus.cc:383] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.637485 1163 raft_consensus.cc:738] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.637734 1186 ts_tablet_manager.cc:1918] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.637691 1163 consensus_queue.cc:260] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.637831 1186 log.cc:1198] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9856e6b625d04f17b0869cee08f69142
I20250112 20:51:44.638115 1186 ts_tablet_manager.cc:1939] T 9856e6b625d04f17b0869cee08f69142 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.638183 1163 ts_tablet_manager.cc:1428] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 20:51:44.638207 1178 tablet_bootstrap.cc:654] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.638437 1163 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.638556 1186 tablet_replica.cc:331] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.638759 1186 raft_consensus.cc:2238] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.638895 1186 raft_consensus.cc:2267] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.639392 1186 ts_tablet_manager.cc:1905] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.639716 1163 tablet_bootstrap.cc:654] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.647450 1164 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.647643 1164 ts_tablet_manager.cc:1397] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250112 20:51:44.647773 2447 ts_tablet_manager.cc:1918] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.647908 2447 log.cc:1198] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/35dcd1a120684446bbe19303fb8416ef
I20250112 20:51:44.648195 2447 ts_tablet_manager.cc:1939] T 35dcd1a120684446bbe19303fb8416ef P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.648587 1164 raft_consensus.cc:357] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.648866 1164 raft_consensus.cc:383] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.648974 1164 raft_consensus.cc:738] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.649302 1164 consensus_queue.cc:260] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.650686 1164 ts_tablet_manager.cc:1428] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:44.651597 1164 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.653174 1164 tablet_bootstrap.cc:654] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.655136 1163 tablet_bootstrap.cc:492] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.655273 1163 ts_tablet_manager.cc:1397] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250112 20:51:44.655952 1163 raft_consensus.cc:357] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.656140 1163 raft_consensus.cc:383] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.656203 1163 raft_consensus.cc:738] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.656425 1163 consensus_queue.cc:260] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.657897 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9856e6b625d04f17b0869cee08f69142 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:44.664387 2208 raft_consensus.cc:491] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.664551 2208 raft_consensus.cc:513] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.665252 2208 leader_election.cc:290] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.666028 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.666230 909 raft_consensus.cc:2463] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.666754 710 leader_election.cc:304] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.667097 2208 raft_consensus.cc:2798] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.667173 2208 raft_consensus.cc:491] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.667244 2208 raft_consensus.cc:3054] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.669703 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.670171 2208 raft_consensus.cc:513] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.671355 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.671358 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.671525 909 raft_consensus.cc:3054] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.672883 1185 ts_tablet_manager.cc:1918] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.672974 1185 log.cc:1198] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:44.673338 1185 ts_tablet_manager.cc:1939] T 2ab635fec8a64efd8e0c527d913ccd7d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.674136 909 raft_consensus.cc:2463] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.674177 1185 tablet_replica.cc:331] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.674355 1185 raft_consensus.cc:2238] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.674528 1185 raft_consensus.cc:2267] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.674772 710 leader_election.cc:304] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.675024 2632 raft_consensus.cc:2798] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.675130 2632 raft_consensus.cc:695] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.675200 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a00f71eb25f84488a78ed3bea61f57e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:44.675348 2632 consensus_queue.cc:237] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.675742 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a00f71eb25f84488a78ed3bea61f57e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.676815 2208 leader_election.cc:290] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.677772 1163 ts_tablet_manager.cc:1428] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250112 20:51:44.678154 1163 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.678870 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:44.679148 449 catalog_manager.cc:5526] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:44.679670 1163 tablet_bootstrap.cc:654] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.679932 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:44.680666 2447 tablet_replica.cc:331] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.680891 2447 raft_consensus.cc:2238] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.681075 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:44.681576 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 89f9b46da06a476ab9f2a93ca74ff736 (table default.Table_a6c9111b483148718ccd2bfb2a54698b [id=b3b49691fa8a41dda2abad8d2f59e423]). Expected version 2 got 0
I20250112 20:51:44.682087 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_8f4ca7a462ed41ff8df87e593847cd3a [id=41df221c4bfb42fe8cd85808401151b2] alter complete (version 2)
I20250112 20:51:44.681051 2447 raft_consensus.cc:2267] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.683144 2447 ts_tablet_manager.cc:1905] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.683239 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_23ec53724daa45fa96c695571826a6ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:44.683694 1186 ts_tablet_manager.cc:1918] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 20:51:44.683744 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23ec53724daa45fa96c695571826a6ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.683812 1186 log.cc:1198] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:44.684125 1186 ts_tablet_manager.cc:1939] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:44.685554 2585 consensus_queue.cc:1035] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.687546 1185 ts_tablet_manager.cc:1905] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.687623 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 2ab635fec8a64efd8e0c527d913ccd7d (table default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]) successfully deleted
I20250112 20:51:44.687981 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.690256 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d0829e9dd5854961b5402b5710624612 (table default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]) successfully deleted
W20250112 20:51:44.690596 710 leader_election.cc:343] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:44.691573 710 consensus_peers.cc:487] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:44.692155 710 leader_election.cc:343] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.692605 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 35dcd1a120684446bbe19303fb8416ef (table default.Table_941f7aeb51a54bf38161efd28eb35340 [id=8d9842a680f543369b38e472a13ead97]) successfully deleted
W20250112 20:51:44.693435 710 leader_election.cc:343] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.695210 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1a4b5ac2f94874b99b3ef790a034fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.696133 1163 tablet_bootstrap.cc:492] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.696310 1163 ts_tablet_manager.cc:1397] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250112 20:51:44.696489 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55686 (request call id 929) took 1035 ms. Trace:
I20250112 20:51:44.696580 452 rpcz_store.cc:276] 0112 20:51:43.660864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.459800 (+798936us) service_pool.cc:225] Handling call
0112 20:51:44.468299 (+ 8499us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14fed03 00007fa2e8601236 00007fa2e8600f65 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:44.505490 (+ 37191us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 20:51:44.506084 (+ 594us) catalog_manager.cc:2122] Created new table and tablet info
0112 20:51:44.696221 (+190137us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 20:51:44.696338 (+ 117us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 20:51:44.696434 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:44.688903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:44.689010 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:44.689018 (+ 8us) write_op.cc:435] Acquired schema lock
0112 20:51:44.689020 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:44.689129 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:44.689141 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:44.689147 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 20:51:44.689218 (+ 71us) tablet.cc:698] Row locks acquired
0112 20:51:44.689221 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:44.689254 (+ 33us) write_op.cc:270] Start()
0112 20:51:44.689287 (+ 33us) write_op.cc:276] Timestamp: P: 1736715104689234 usec, L: 0
0112 20:51:44.689293 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:44.689451 (+ 158us) log.cc:844] Serialized 433 byte log entry
0112 20:51:44.695398 (+ 5947us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.695542 (+ 144us) write_op.cc:301] APPLY: starting
0112 20:51:44.695557 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:44.695594 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:44.695619 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:44.695719 (+ 100us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:44.695751 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 20:51:44.695761 (+ 10us) write_op.cc:312] APPLY: finished
0112 20:51:44.695823 (+ 62us) log.cc:844] Serialized 29 byte log entry
0112 20:51:44.695890 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.695907 (+ 17us) write_op.cc:454] Released schema lock
0112 20:51:44.695915 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 20:51:44.695984 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.695988 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18788736,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":434,"num_ops":2,"prepare.queue_time_us":177546,"prepare.run_cpu_time_us":706,"prepare.run_wall_time_us":4931,"raft.queue_time_us":926,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":366,"replication_time_us":6074}]]}
I20250112 20:51:44.697085 1163 raft_consensus.cc:357] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.697212 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9219fc971c4d15b0d383b20b61e540 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:30 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.697270 1163 raft_consensus.cc:383] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.697324 1163 raft_consensus.cc:738] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.697412 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.697571 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet bc05d287d81c46558acdd9a42edda6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.697512 1163 consensus_queue.cc:260] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.698001 1163 ts_tablet_manager.cc:1428] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:44.698276 1163 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.699352 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet eaa091fe2d3c4f72815f228ac3e39128
I20250112 20:51:44.699592 1163 tablet_bootstrap.cc:654] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.699961 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:44.700517 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d97953504d9648629174944c086c84ef" } new_table_name: "default.Table_a22256f4acae465f99ea64d504440168" modify_external_catalogs: true
I20250112 20:51:44.700680 2447 ts_tablet_manager.cc:1918] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.700803 2447 log.cc:1198] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/2469127891524be5ac3dc13fbcf35c7a
I20250112 20:51:44.701076 2447 ts_tablet_manager.cc:1939] T 2469127891524be5ac3dc13fbcf35c7a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.701907 2447 tablet_replica.cc:331] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.703048 884 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.703486 737 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.704277 1042 raft_consensus.cc:3054] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.702068 2447 raft_consensus.cc:2238] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.705205 2447 raft_consensus.cc:2267] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.705669 2447 ts_tablet_manager.cc:1905] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.706974 1164 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.707098 1164 ts_tablet_manager.cc:1397] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.004s
I20250112 20:51:44.711987 1163 tablet_bootstrap.cc:492] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.712136 1163 ts_tablet_manager.cc:1397] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20250112 20:51:44.712291 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 2469127891524be5ac3dc13fbcf35c7a (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
I20250112 20:51:44.712769 1163 raft_consensus.cc:357] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.712960 1163 raft_consensus.cc:383] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.713014 1163 raft_consensus.cc:738] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.713213 1163 consensus_queue.cc:260] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.713698 1163 ts_tablet_manager.cc:1428] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:44.714162 1163 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.715073 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa091fe2d3c4f72815f228ac3e39128 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.715447 1163 tablet_bootstrap.cc:654] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.707708 1164 raft_consensus.cc:357] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.716996 1164 raft_consensus.cc:383] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.719123 1164 raft_consensus.cc:738] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.719408 1164 consensus_queue.cc:260] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.720178 1164 ts_tablet_manager.cc:1428] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 20:51:44.721727 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b299e16586145ee9ed5a7b118c049b3
I20250112 20:51:44.722442 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.722992 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.723582 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b299e16586145ee9ed5a7b118c049b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.725744 2447 ts_tablet_manager.cc:1918] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.725867 2447 log.cc:1198] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d570daec845f40778c48ff3f2e2e30cb
I20250112 20:51:44.726169 2447 ts_tablet_manager.cc:1939] T d570daec845f40778c48ff3f2e2e30cb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.726989 2447 tablet_replica.cc:331] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.727094 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d570daec845f40778c48ff3f2e2e30cb (table default.Table_56e32a726bf14d0193beb81aef87a2de [id=f2b308db7f704aedb27aedae971d1ec9]) successfully deleted
I20250112 20:51:44.727183 2447 raft_consensus.cc:2238] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.727304 2447 raft_consensus.cc:2267] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.727807 2447 ts_tablet_manager.cc:1905] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.731786 1163 tablet_bootstrap.cc:492] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.731905 1163 ts_tablet_manager.cc:1397] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250112 20:51:44.732493 1163 raft_consensus.cc:357] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.732646 1163 raft_consensus.cc:383] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.732686 1163 raft_consensus.cc:738] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.733327 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65cffe4814de46449661f915c2371007" } modify_external_catalogs: true
I20250112 20:51:44.733448 2447 ts_tablet_manager.cc:1918] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.733000 1163 consensus_queue.cc:260] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.733530 2447 log.cc:1198] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f3ff8ac6f4d94c10b03a764bbf910c44
I20250112 20:51:44.733505 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65cffe4814de46449661f915c2371007" } modify_external_catalogs: true
I20250112 20:51:44.733867 2447 ts_tablet_manager.cc:1939] T f3ff8ac6f4d94c10b03a764bbf910c44 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.735930 1163 ts_tablet_manager.cc:1428] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:44.737493 2447 tablet_replica.cc:331] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.737591 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f3ff8ac6f4d94c10b03a764bbf910c44 (table default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]) successfully deleted
I20250112 20:51:44.737675 2447 raft_consensus.cc:2238] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.737794 2447 raft_consensus.cc:2267] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.738255 2447 ts_tablet_manager.cc:1905] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.738615 1178 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.738761 1178 ts_tablet_manager.cc:1397] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.013s
I20250112 20:51:44.739648 1163 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.739555 1178 raft_consensus.cc:357] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.739851 1178 raft_consensus.cc:383] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.739953 1178 raft_consensus.cc:738] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.740245 1178 consensus_queue.cc:260] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.740800 1178 ts_tablet_manager.cc:1428] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.741073 1178 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.741303 1163 tablet_bootstrap.cc:654] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.742753 1178 tablet_bootstrap.cc:654] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.743000 1185 ts_tablet_manager.cc:1918] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.743111 1185 log.cc:1198] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4561956c97824824a55283a8da21929e
I20250112 20:51:44.743422 1185 ts_tablet_manager.cc:1939] T 4561956c97824824a55283a8da21929e P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.747345 2632 raft_consensus.cc:491] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.747431 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 30156185f6104eada87b936fc1829df9
I20250112 20:51:44.747488 2632 raft_consensus.cc:513] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.748092 2632 leader_election.cc:290] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.748136 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_7afee759e98341ce855fc733cb16a5f9" } new_table_name: "default.Table_9a3f2d4ebf3242f981729043ca9a3470" modify_external_catalogs: true
I20250112 20:51:44.748795 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.748979 912 raft_consensus.cc:2463] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.749667 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (table default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]) successfully deleted
I20250112 20:51:44.749641 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.749985 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 30156185f6104eada87b936fc1829df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.750283 710 leader_election.cc:304] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.750345 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 30156185f6104eada87b936fc1829df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.750545 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 30156185f6104eada87b936fc1829df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:44.750818 710 leader_election.cc:343] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.751119 2632 raft_consensus.cc:2798] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.751194 2632 raft_consensus.cc:491] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.751262 2632 raft_consensus.cc:3054] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.754161 2632 raft_consensus.cc:513] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.755123 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.755838 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8098069639a2435ab500908db9ecd12a
I20250112 20:51:44.759342 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8098069639a2435ab500908db9ecd12a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.759346 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8098069639a2435ab500908db9ecd12a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:44.759631 710 leader_election.cc:343] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.759670 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 8098069639a2435ab500908db9ecd12a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.760396 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.760567 912 raft_consensus.cc:3054] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.768136 2563 raft_consensus.cc:491] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.768345 2208 raft_consensus.cc:491] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.768464 2208 raft_consensus.cc:513] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.769156 2208 leader_election.cc:290] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.770100 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.770290 908 raft_consensus.cc:2463] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.771127 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:44.768409 2563 raft_consensus.cc:513] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.771867 710 leader_election.cc:343] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.771888 2563 leader_election.cc:290] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.772058 710 leader_election.cc:304] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:44.772534 2632 leader_election.cc:290] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.772743 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.773342 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fac2cf43c24c158e340cbc8dd176c3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.773533 912 raft_consensus.cc:2463] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.774165 983 leader_election.cc:304] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.774358 2208 raft_consensus.cc:2798] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.774430 2208 raft_consensus.cc:491] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.774492 2208 raft_consensus.cc:3054] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.774658 2563 raft_consensus.cc:2743] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.755190 2447 ts_tablet_manager.cc:1918] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.775192 2447 log.cc:1198] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/70653110aec74727acec9d63c2eb0f05
I20250112 20:51:44.775493 2447 ts_tablet_manager.cc:1939] T 70653110aec74727acec9d63c2eb0f05 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.776316 2447 tablet_replica.cc:331] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.776499 1186 tablet_replica.cc:331] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:44.776510 2447 raft_consensus.cc:2238] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.776623 2447 raft_consensus.cc:2267] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.776862 1186 raft_consensus.cc:2238] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.777206 1186 raft_consensus.cc:2267] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.777292 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 70653110aec74727acec9d63c2eb0f05 (table default.Table_8284ddd01ebb44d49bfeac291415f6b1 [id=b4cca7b35ba14cf7bb12e84c441387e3]) successfully deleted
I20250112 20:51:44.777833 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_54fe2ed58a8640d3abeb4b662e75a753"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:44.777947 710 leader_election.cc:304] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:44.778103 1186 ts_tablet_manager.cc:1905] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:44.778378 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54fe2ed58a8640d3abeb4b662e75a753 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.778421 2585 raft_consensus.cc:2798] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.778524 2585 raft_consensus.cc:695] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.778662 2208 raft_consensus.cc:513] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.778766 2585 consensus_queue.cc:237] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.779217 2208 leader_election.cc:290] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.779346 2208 consensus_queue.cc:1035] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.779804 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9c771feed27441ec8340f7279a4a06c1" } new_table_name: "default.Table_13fd362e79604cebb738b3c33dd03cc6" modify_external_catalogs: true
I20250112 20:51:44.780361 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.780517 908 raft_consensus.cc:3054] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.781188 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_90428f7ce05e4667a57dc6fec766a9be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.781656 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90428f7ce05e4667a57dc6fec766a9be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.783322 908 raft_consensus.cc:2463] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.783913 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.783937 1042 raft_consensus.cc:3054] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.784454 710 leader_election.cc:304] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.784699 2632 raft_consensus.cc:2798] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:44.784795 2632 raft_consensus.cc:695] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.784951 2447 ts_tablet_manager.cc:1905] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.785190 2563 raft_consensus.cc:491] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.785306 2563 raft_consensus.cc:513] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.785913 2563 leader_election.cc:290] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.785032 2632 consensus_queue.cc:237] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.788748 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 29c4b2cf79d9424eaa2ae8169b4d22a5
W20250112 20:51:44.789772 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c058dcab55ca49298ff544dcb8c3f754 consensus state has no leader
I20250112 20:51:44.790203 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f284beef9d64af2a22525800952a18b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.790887 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f284beef9d64af2a22525800952a18b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.794241 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2f5001cff98b4cb3afa8fae9e15f2fbb" } new_table_name: "default.Table_12a742837d8f490f8185ae12438e2b43" modify_external_catalogs: true
I20250112 20:51:44.795712 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:44.796643 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b040a8eefbe44fd49f6cf0bd2fdcf651 consensus state has no leader
I20250112 20:51:44.796991 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 3dc719cd84124082b782016011e37c8b from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:44.798256 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.798444 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.799273 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c229ef75fe0c46cd9f2cd6817f5b2a29" } new_table_name: "default.Table_3fe5e44f1c6d4e1ea61b10d21397bb62" modify_external_catalogs: true
W20250112 20:51:44.801208 710 leader_election.cc:343] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.801527 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.802611 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2513dfbdccf9426ba98711dae084e5ef" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.803213 983 leader_election.cc:304] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.803622 2563 raft_consensus.cc:2743] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.804240 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_35d70052f03445b3be247723d56eec3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.804721 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d70052f03445b3be247723d56eec3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.816323 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fdbdfd9f44f6454c9839477f1f437558"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.817153 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdbdfd9f44f6454c9839477f1f437558 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:44.826399 710 consensus_peers.cc:487] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6092a31de9bc47e4bae5f5aebb180c6d. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.827116 2632 raft_consensus.cc:491] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.827561 2632 raft_consensus.cc:513] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.832980 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.833612 710 leader_election.cc:343] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.838690 2447 ts_tablet_manager.cc:1918] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.838833 2447 log.cc:1198] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3bbb679585fc44699553fe85ea044566
I20250112 20:51:44.839221 2447 ts_tablet_manager.cc:1939] T 3bbb679585fc44699553fe85ea044566 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.849370 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:44.849658 909 raft_consensus.cc:2463] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:44.850127 710 leader_election.cc:304] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:44.851234 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 816) took 1021 ms. Trace:
I20250112 20:51:44.851321 2447 rpcz_store.cc:276] 0112 20:51:43.829448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.829486 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:44.776270 (+946784us) ts_tablet_manager.cc:1145] Deleting tablet 3bbb679585fc44699553fe85ea044566
0112 20:51:44.776279 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.819453 (+ 43174us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.838568 (+ 19115us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.851212 (+ 12644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":946688}
I20250112 20:51:44.851753 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3bbb679585fc44699553fe85ea044566 (table default.Table_8eeab516a6794571a8336068218e7a3a [id=2bc2a6f29dda4e13a781af77ba66b46f]) successfully deleted
W20250112 20:51:44.852515 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5311cd2a18654994a2fdb88be3e9490b consensus state has no leader
I20250112 20:51:44.852757 2747 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.852922 2613 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.853266 2585 raft_consensus.cc:2798] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:44.853354 2585 raft_consensus.cc:491] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.853412 2585 raft_consensus.cc:3054] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.855906 2585 raft_consensus.cc:513] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.857461 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:44.857627 908 raft_consensus.cc:3054] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.857465 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:44.859603 2447 tablet_replica.cc:331] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.859777 2447 raft_consensus.cc:2238] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.859898 2447 raft_consensus.cc:2267] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.860188 908 raft_consensus.cc:2463] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:44.860373 2447 ts_tablet_manager.cc:1905] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.861516 710 leader_election.cc:304] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:44.861877 2059 raft_consensus.cc:2798] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:44.861939 710 leader_election.cc:343] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.861987 2059 raft_consensus.cc:695] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:44.862226 2059 consensus_queue.cc:237] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
W20250112 20:51:44.867846 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d153d6ce727a4819ba9c409f9b87552e consensus state has no leader
I20250112 20:51:44.873595 1178 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.873811 1178 ts_tablet_manager.cc:1397] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.007s
I20250112 20:51:44.874665 1178 raft_consensus.cc:357] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.875005 1178 raft_consensus.cc:383] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.875140 1178 raft_consensus.cc:738] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.875423 1178 consensus_queue.cc:260] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.877630 1178 ts_tablet_manager.cc:1428] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:44.878671 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d4025212c78a43f2af864cca35e42251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.879303 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4025212c78a43f2af864cca35e42251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.879724 2585 leader_election.cc:290] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.886624 1185 tablet_replica.cc:331] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.886816 1185 raft_consensus.cc:2238] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:44.887108 1185 raft_consensus.cc:2267] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.887622 1185 ts_tablet_manager.cc:1905] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.894631 1185 ts_tablet_manager.cc:1918] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.894762 1185 log.cc:1198] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:44.895233 1185 ts_tablet_manager.cc:1939] T 6092a31de9bc47e4bae5f5aebb180c6d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:44.896286 1185 tablet_replica.cc:331] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:44.896514 1185 raft_consensus.cc:2238] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.896642 1185 raft_consensus.cc:2267] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.897619 1164 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:44.897982 2632 leader_election.cc:290] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.898005 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4561956c97824824a55283a8da21929e (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
I20250112 20:51:44.898520 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 6092a31de9bc47e4bae5f5aebb180c6d (table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]) successfully deleted
I20250112 20:51:44.899420 1164 tablet_bootstrap.cc:654] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.899585 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
/tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--webserver_interface=127.0.104.61
--webserver_port=39277
--builtin_ntp_servers=127.0.104.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 20:51:44.899984 2447 ts_tablet_manager.cc:1918] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:44.900090 2447 log.cc:1198] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4f34aa32d6474561988b2cf3d0e8fd3b
I20250112 20:51:44.900380 2447 ts_tablet_manager.cc:1939] T 4f34aa32d6474561988b2cf3d0e8fd3b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:44.905570 2563 raft_consensus.cc:491] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.905746 2563 raft_consensus.cc:513] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:44.906468 2563 leader_election.cc:290] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:44.906592 452 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:44.907549 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_0466e8f4f9b14dc8b813c109373e1d1e [id=9dda94626ac842fabfa576cfb417dac1] alter complete (version 3)
W20250112 20:51:44.907546 983 leader_election.cc:343] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:44.908986 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 815) took 1079 ms. Trace:
I20250112 20:51:44.909086 2447 rpcz_store.cc:276] 0112 20:51:43.829354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.829595 (+ 241us) service_pool.cc:225] Handling call
0112 20:51:44.859557 (+1029962us) ts_tablet_manager.cc:1145] Deleting tablet 4f34aa32d6474561988b2cf3d0e8fd3b
0112 20:51:44.859567 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.863156 (+ 3589us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.899872 (+ 36716us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.908964 (+ 9092us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029868}
I20250112 20:51:44.910749 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4f34aa32d6474561988b2cf3d0e8fd3b (table default.Table_ef05483d74ba466ea37bf2e84489fc15 [id=3ddc179c72f74f78b0ec696d624429c4]) successfully deleted
I20250112 20:51:44.911726 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.911756 881 tablet_service.cc:1467] Processing CreateTablet for tablet a6a75f8dc80a46dcbf17bd838b9a4145 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.911996 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.912142 875 tablet_service.cc:1467] Processing CreateTablet for tablet 5797c2fd8e21490ca00ef171965c31ad (DEFAULT_TABLE table=default.Table_8f0fb6214ee84e8a88e5292e5db7756d [id=a63958fea1044a83ae2e91f329da462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.912164 884 tablet_service.cc:1467] Processing CreateTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.912374 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.912520 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.913213 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.913617 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.914184 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.914337 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.914451 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.915591 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.915660 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.917030 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a6a75f8dc80a46dcbf17bd838b9a4145 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.917400 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a75f8dc80a46dcbf17bd838b9a4145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.917374 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a70d5e6e49734f619ecedc6196b04c1d (DEFAULT_TABLE table=default.Table_b403df49b35f4c1db5345f2e317fc890 [id=799cc8ef5c3b4011b2a4de21513fa73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.917637 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d5e6e49734f619ecedc6196b04c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.917857 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 5797c2fd8e21490ca00ef171965c31ad (DEFAULT_TABLE table=default.Table_8f0fb6214ee84e8a88e5292e5db7756d [id=a63958fea1044a83ae2e91f329da462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.918159 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5797c2fd8e21490ca00ef171965c31ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:44.918123 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cf6d73ab981c4e87aa229a49fb3b6913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d73ab981c4e87aa229a49fb3b6913 already in progress: creating tablet
I20250112 20:51:44.920889 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.921419 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 903027fe49e34a308108356ba89a9c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.922515 737 tablet_service.cc:1467] Processing CreateTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.922514 754 tablet_service.cc:1467] Processing CreateTablet for tablet a6a75f8dc80a46dcbf17bd838b9a4145 (DEFAULT_TABLE table=default.Table_c53e401a48884786b8e51ae3d66e8d87 [id=dff0f9c81e66471083454d01428561ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.922963 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.923125 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.923300 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a75f8dc80a46dcbf17bd838b9a4145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.926074 1178 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:44.928403 1178 tablet_bootstrap.cc:654] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:44.928468 885 tablet_service.cc:1467] Processing CreateTablet for tablet a70d5e6e49734f619ecedc6196b04c1d (DEFAULT_TABLE table=default.Table_b403df49b35f4c1db5345f2e317fc890 [id=799cc8ef5c3b4011b2a4de21513fa73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.928551 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a75f8dc80a46dcbf17bd838b9a4145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.928824 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d5e6e49734f619ecedc6196b04c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.929037 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5797c2fd8e21490ca00ef171965c31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.929543 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2e8ea09f0c2c4ccea9540106b863d8a8" } modify_external_catalogs: true
I20250112 20:51:44.929611 892 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.929734 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2e8ea09f0c2c4ccea9540106b863d8a8" } modify_external_catalogs: true
I20250112 20:51:44.931280 1185 ts_tablet_manager.cc:1905] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.932977 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.937001 1042 raft_consensus.cc:3054] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.939441 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e9a8102300fb471f93b5e5ba18c2f532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.940007 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9a8102300fb471f93b5e5ba18c2f532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.940045 2108 consensus_queue.cc:1035] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.936450 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:44.941370 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:44.941644 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:44.942059 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:44.945907 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:44.946996 1163 tablet_bootstrap.cc:492] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.947141 1163 ts_tablet_manager.cc:1397] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.208s user 0.006s sys 0.018s
I20250112 20:51:44.947705 737 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.948002 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5797c2fd8e21490ca00ef171965c31ad (DEFAULT_TABLE table=default.Table_8f0fb6214ee84e8a88e5292e5db7756d [id=a63958fea1044a83ae2e91f329da462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.948145 1163 raft_consensus.cc:357] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.948315 1163 raft_consensus.cc:383] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.948364 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5797c2fd8e21490ca00ef171965c31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.948364 1163 raft_consensus.cc:738] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.948551 1163 consensus_queue.cc:260] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.949059 1163 ts_tablet_manager.cc:1428] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.949293 1163 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.950807 1163 tablet_bootstrap.cc:654] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:44.951651 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c23f5f085dd84d68beb277676225e7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23f5f085dd84d68beb277676225e7e2 already in progress: creating tablet
I20250112 20:51:44.951920 2447 tablet_replica.cc:331] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:44.952126 2447 raft_consensus.cc:2238] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:44.952258 2447 raft_consensus.cc:2267] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:44.952708 2447 ts_tablet_manager.cc:1905] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:44.960165 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a6c9111b483148718ccd2bfb2a54698b" } modify_external_catalogs: true
I20250112 20:51:44.960376 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a6c9111b483148718ccd2bfb2a54698b" } modify_external_catalogs: true
I20250112 20:51:44.961238 1178 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:44.961352 1178 ts_tablet_manager.cc:1397] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20250112 20:51:44.962381 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.962378 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b5e7418fef2d44709ff22f9961666ee3" } modify_external_catalogs: true
I20250112 20:51:44.962538 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b5e7418fef2d44709ff22f9961666ee3" } modify_external_catalogs: true
I20250112 20:51:44.963034 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab635fec8a64efd8e0c527d913ccd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963223 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963367 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ab0ae10f09469fa4f913e049002481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963500 753 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963557 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963629 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963972 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.962082 1178 raft_consensus.cc:357] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.964140 737 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.964180 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6092a31de9bc47e4bae5f5aebb180c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:31 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.963941 753 tablet_service.cc:1467] Processing CreateTablet for tablet a70d5e6e49734f619ecedc6196b04c1d (DEFAULT_TABLE table=default.Table_b403df49b35f4c1db5345f2e317fc890 [id=799cc8ef5c3b4011b2a4de21513fa73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:44.964495 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d5e6e49734f619ecedc6196b04c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:44.968361 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3dc719cd84124082b782016011e37c8b
I20250112 20:51:44.968848 455 catalog_manager.cc:6596] ReplaceTablet: tablet 3dc719cd84124082b782016011e37c8b deleted and replaced by tablet 282eeeb143bc40d899d6c25223582f24
I20250112 20:51:44.969682 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dc719cd84124082b782016011e37c8b with delete_type TABLET_DATA_DELETED (replaced by tablet 282eeeb143bc40d899d6c25223582f24) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:44.969942 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dc719cd84124082b782016011e37c8b with delete_type TABLET_DATA_DELETED (replaced by tablet 282eeeb143bc40d899d6c25223582f24) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.970019 1178 raft_consensus.cc:383] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.970099 1178 raft_consensus.cc:738] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:44.958798 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 89238.7us
I20250112 20:51:44.972643 2108 raft_consensus.cc:491] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:44.972764 2108 raft_consensus.cc:513] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.973374 2108 leader_election.cc:290] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:44.973807 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:44.974316 847 leader_election.cc:343] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:44.974939 1044 raft_consensus.cc:3054] T 2513dfbdccf9426ba98711dae084e5ef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:44.975244 1178 consensus_queue.cc:260] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.976346 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dc719cd84124082b782016011e37c8b with delete_type TABLET_DATA_DELETED (replaced by tablet 282eeeb143bc40d899d6c25223582f24) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.976537 2751 tablet.cc:1721] T 4390c60d6c724190a216ef9c961b9592 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:44.979321 2208 consensus_queue.cc:1035] T 2513dfbdccf9426ba98711dae084e5ef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:44.981280 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cc9d5d182827465abd3799d3ad9a6484
I20250112 20:51:44.982164 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d5d182827465abd3799d3ad9a6484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:44.982200 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d5d182827465abd3799d3ad9a6484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:44.982462 885 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d5d182827465abd3799d3ad9a6484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:44.982892 983 leader_election.cc:343] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:44.983028 983 leader_election.cc:304] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.983479 2563 raft_consensus.cc:2743] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.988085 1164 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:44.988379 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d072ebae78e40c48e158eb65775476c from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:44.988246 1164 ts_tablet_manager.cc:1397] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.002s
I20250112 20:51:44.989567 1186 ts_tablet_manager.cc:1918] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:44.990146 1164 raft_consensus.cc:357] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.990336 1164 raft_consensus.cc:383] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.990393 1164 raft_consensus.cc:738] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.990622 1164 consensus_queue.cc:260] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:44.991773 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_92f355d4058f40b6b0979ff3f3f1af14" } modify_external_catalogs: true
I20250112 20:51:44.992028 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_92f355d4058f40b6b0979ff3f3f1af14" } modify_external_catalogs: true
I20250112 20:51:44.992303 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:44.993072 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_febe7e80fab0422da18aa05a5609d2f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.993614 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_febe7e80fab0422da18aa05a5609d2f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.993791 1163 tablet_bootstrap.cc:492] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:44.993912 1163 ts_tablet_manager.cc:1397] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
W20250112 20:51:44.992316 710 consensus_peers.cc:487] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:44.994576 1163 raft_consensus.cc:357] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.994755 1163 raft_consensus.cc:383] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:44.994812 1163 raft_consensus.cc:738] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:44.995020 1163 consensus_queue.cc:260] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:44.995126 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_53a9e79cf0434f96bdffc0b5c7f26934"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:44.994798 847 leader_election.cc:343] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:44.995601 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53a9e79cf0434f96bdffc0b5c7f26934 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:44.995824 1163 ts_tablet_manager.cc:1428] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:44.996155 1163 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:44.976871 1178 ts_tablet_manager.cc:1428] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250112 20:51:44.995375 847 leader_election.cc:304] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:44.998000 2521 raft_consensus.cc:2743] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:44.999150 1186 log.cc:1198] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4561956c97824824a55283a8da21929e
I20250112 20:51:44.999673 1186 ts_tablet_manager.cc:1939] T 4561956c97824824a55283a8da21929e P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:45.005242 1163 tablet_bootstrap.cc:654] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.007467 2447 ts_tablet_manager.cc:1918] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.008183 1178 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.008705 2447 log.cc:1198] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8b7b788acf8047acb34ca2596bbf141b
I20250112 20:51:45.009085 2447 ts_tablet_manager.cc:1939] T 8b7b788acf8047acb34ca2596bbf141b P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.010085 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8b7b788acf8047acb34ca2596bbf141b (table default.Table_c09ffcd01d594a1d8b065295cf69258a [id=22e212eacf0449b48069d0c9b7cda461]) successfully deleted
I20250112 20:51:45.012179 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 853) took 1210 ms. Trace:
I20250112 20:51:45.012311 1186 rpcz_store.cc:276] 0112 20:51:43.801973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.802023 (+ 50us) service_pool.cc:225] Handling call
0112 20:51:44.748615 (+946592us) ts_tablet_manager.cc:1145] Deleting tablet 4561956c97824824a55283a8da21929e
0112 20:51:44.776445 (+ 27830us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:44.826866 (+ 50421us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:44.989466 (+162600us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.012163 (+ 22697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":946465}
I20250112 20:51:45.013451 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f8b3eae7e0294abfa7f8f431ff6aee97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.013996 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.018245 1185 ts_tablet_manager.cc:1918] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.018618 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 846) took 1217 ms. Trace:
I20250112 20:51:45.018677 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4561956c97824824a55283a8da21929e (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
I20250112 20:51:45.022390 1186 rpcz_store.cc:276] 0112 20:51:43.800751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.819067 (+ 18316us) service_pool.cc:225] Handling call
0112 20:51:45.018497 (+1199430us) ts_tablet_manager.cc:1145] Deleting tablet 1748fb558c4d48f49432840055d71c67
0112 20:51:45.018507 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.018601 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199209}
I20250112 20:51:45.022845 1186 tablet_replica.cc:331] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:45.022800 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1748fb558c4d48f49432840055d71c67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1748fb558c4d48f49432840055d71c67 already in progress: creating tablet
I20250112 20:51:45.023015 1186 raft_consensus.cc:2238] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.023170 1186 raft_consensus.cc:2267] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:45.027443 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c45ac747ab7f4990a03a39553424e951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c45ac747ab7f4990a03a39553424e951 already in progress: creating tablet
W20250112 20:51:45.028082 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9498a04f20dd4cf4a283f17d37ca49ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9498a04f20dd4cf4a283f17d37ca49ec already in progress: creating tablet
I20250112 20:51:45.028703 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_13fd362e79604cebb738b3c33dd03cc6 [id=2ee1f97fef0f4e2b83932132d0936504] alter complete (version 3)
I20250112 20:51:45.021811 1178 tablet_bootstrap.cc:654] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:45.029758 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d6a0679bf1ed4bbc85f14f62dfcc114d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a0679bf1ed4bbc85f14f62dfcc114d already in progress: creating tablet
I20250112 20:51:45.030035 2447 tablet_replica.cc:331] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.030241 2447 raft_consensus.cc:2238] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.030387 2447 raft_consensus.cc:2267] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.030848 2447 ts_tablet_manager.cc:1905] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.038491 1163 tablet_bootstrap.cc:492] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.038631 1163 ts_tablet_manager.cc:1397] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250112 20:51:45.039357 1163 raft_consensus.cc:357] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.039574 1163 raft_consensus.cc:383] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.039639 1163 raft_consensus.cc:738] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.039884 1163 consensus_queue.cc:260] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.040719 1163 ts_tablet_manager.cc:1428] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 20:51:45.041003 1163 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.043764 1163 tablet_bootstrap.cc:654] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.045295 1185 log.cc:1198] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/903027fe49e34a308108356ba89a9c3a
I20250112 20:51:45.045557 2208 raft_consensus.cc:491] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.045637 1185 ts_tablet_manager.cc:1939] T 903027fe49e34a308108356ba89a9c3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:45.045692 2208 raft_consensus.cc:513] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.046258 2208 leader_election.cc:290] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:45.046391 1186 ts_tablet_manager.cc:1905] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.046669 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 903027fe49e34a308108356ba89a9c3a (table default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]) successfully deleted
I20250112 20:51:45.047132 2447 ts_tablet_manager.cc:1918] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.047176 1185 tablet_replica.cc:331] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:45.047240 2447 log.cc:1198] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/ad1a4b5ac2f94874b99b3ef790a034fd
I20250112 20:51:45.047356 1185 raft_consensus.cc:2238] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:45.047365 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c45ac747ab7f4990a03a39553424e951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c45ac747ab7f4990a03a39553424e951 already in progress: creating tablet
I20250112 20:51:45.047524 1185 raft_consensus.cc:2267] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.047533 2447 ts_tablet_manager.cc:1939] T ad1a4b5ac2f94874b99b3ef790a034fd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:45.047595 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9498a04f20dd4cf4a283f17d37ca49ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9498a04f20dd4cf4a283f17d37ca49ec already in progress: creating tablet
W20250112 20:51:45.047881 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d6a0679bf1ed4bbc85f14f62dfcc114d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a0679bf1ed4bbc85f14f62dfcc114d already in progress: creating tablet
I20250112 20:51:45.048071 1185 ts_tablet_manager.cc:1905] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:45.048094 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet eaa091fe2d3c4f72815f228ac3e39128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaa091fe2d3c4f72815f228ac3e39128 already in progress: creating tablet
W20250112 20:51:45.048342 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b299e16586145ee9ed5a7b118c049b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b299e16586145ee9ed5a7b118c049b3 already in progress: creating tablet
I20250112 20:51:45.022303 1164 ts_tablet_manager.cc:1428] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250112 20:51:45.053107 1164 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.053397 1044 raft_consensus.cc:3054] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.056499 1164 tablet_bootstrap.cc:654] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.059136 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d80791d8582446b89adf212d3d6507eb
I20250112 20:51:45.060470 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.060650 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.061025 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.062711 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.062800 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.062705 885 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.062979 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.063055 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:45.063181 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.063241 909 raft_consensus.cc:2463] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:45.063537 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.065961 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02" } modify_external_catalogs: true
I20250112 20:51:45.066152 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fc24e0db0d3408eb5aa7ae1ba641d02" } modify_external_catalogs: true
I20250112 20:51:45.066226 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.067296 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_800fbaf988064a7e8810c2617a6c8863" } new_table_name: "default.Table_b0f906299bab48059b43f99eee14bc49" modify_external_catalogs: true
I20250112 20:51:45.068346 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 4561956c97824824a55283a8da21929e with delete_type TABLET_DATA_DELETED (replaced by tablet d80791d8582446b89adf212d3d6507eb) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.068639 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d706f88f14ab4fd9a459944863afaacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.068614 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1e8f92a0b14c40829a4e275e8327e838" } new_table_name: "default.Table_316b86d7610444219e7d6f52c7d19556" modify_external_catalogs: true
I20250112 20:51:45.068912 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:45.069705 710 leader_election.cc:343] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.071245 710 leader_election.cc:304] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.071830 2208 raft_consensus.cc:2798] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.071902 2208 raft_consensus.cc:491] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.071960 2208 raft_consensus.cc:3054] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.073223 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:45.074640 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_acdb4e459bbd4e91bc58571fee9062f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.075275 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d153d6ce727a4819ba9c409f9b87552e (table default.Table_3fe5e44f1c6d4e1ea61b10d21397bb62 [id=e084d844ac1048799cd7333e1f5530cd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d153d6ce727a4819ba9c409f9b87552e consensus state has no leader
W20250112 20:51:45.075472 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acdb4e459bbd4e91bc58571fee9062f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.075459 2208 raft_consensus.cc:513] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.076217 2208 leader_election.cc:290] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:45.076742 2748 consensus_queue.cc:1035] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.081772 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e8bea4d83ea644a0930a29f61e7a5fc8
I20250112 20:51:45.082482 2563 raft_consensus.cc:491] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.082576 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.082613 2563 raft_consensus.cc:513] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.083340 2563 leader_election.cc:290] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.085575 2447 tablet_replica.cc:331] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.086231 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet ad1a4b5ac2f94874b99b3ef790a034fd (table default.Table_a64579137f154510b4d9b3432f9b3403 [id=43e40215eb4341788b44e6e1999c0ffa]) successfully deleted
I20250112 20:51:45.086333 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.086508 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2ce3eaab80fa47539dc4e0fd79c1c80b" } new_table_name: "default.Table_f75dca2565be402990f84a4a1afe5179" modify_external_catalogs: true
I20250112 20:51:45.087528 2521 consensus_queue.cc:1035] T 5f9b7d9218094a63ad5bad03f0d27eb5 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.085749 2447 raft_consensus.cc:2238] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.088052 2447 raft_consensus.cc:2267] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:45.087110 983 leader_election.cc:343] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 54ab0ae10f09469fa4f913e049002481
I20250112 20:51:45.086222 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.088526 2447 ts_tablet_manager.cc:1905] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.090489 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:45.090687 909 raft_consensus.cc:3054] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.092690 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 3f6ad2ac50c745448031bdf94e0eea59 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.093271 909 raft_consensus.cc:2463] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:45.094225 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:45.097385 710 leader_election.cc:343] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.102844 1164 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:45.102989 1164 ts_tablet_manager.cc:1397] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.003s
I20250112 20:51:45.111161 2748 consensus_queue.cc:1035] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.111354 710 leader_election.cc:304] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.111881 2748 raft_consensus.cc:2798] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.112001 2748 raft_consensus.cc:695] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:45.112218 2748 consensus_queue.cc:237] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.121317 2447 ts_tablet_manager.cc:1918] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.121433 2447 log.cc:1198] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/aa9219fc971c4d15b0d383b20b61e540
I20250112 20:51:45.121728 2447 ts_tablet_manager.cc:1939] T aa9219fc971c4d15b0d383b20b61e540 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.127534 1163 tablet_bootstrap.cc:492] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.127678 1163 ts_tablet_manager.cc:1397] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.002s
I20250112 20:51:45.129966 1178 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.130098 1178 ts_tablet_manager.cc:1397] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.000s
I20250112 20:51:45.130836 1178 raft_consensus.cc:357] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.131021 1178 raft_consensus.cc:383] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.131078 1178 raft_consensus.cc:738] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.133638 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet aa9219fc971c4d15b0d383b20b61e540 (table default.Table_70dd4d25b3854472992074db0c6fbeb3 [id=841e7960e3ad4dccb5c293a07a284b7b]) successfully deleted
I20250112 20:51:45.134137 1163 raft_consensus.cc:357] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.134337 1163 raft_consensus.cc:383] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.134385 1163 raft_consensus.cc:738] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.135409 452 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:45.136974 1186 ts_tablet_manager.cc:1918] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:45.137085 1186 log.cc:1198] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:45.137282 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet d7117edf078848c0a1c83ab8e804dfed with delete_type TABLET_DATA_DELETED (replaced by tablet e2c41c2c92b442b986d6b75708c7420c) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.137665 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ea686bd946c94aeca2db6212d2fa6b7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.138320 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea686bd946c94aeca2db6212d2fa6b7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.140141 2447 tablet_replica.cc:331] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.140367 2447 raft_consensus.cc:2238] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.140511 2447 raft_consensus.cc:2267] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.140558 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 9856e6b625d04f17b0869cee08f69142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:40 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.141471 1164 raft_consensus.cc:357] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:45.141654 983 leader_election.cc:343] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 54ab0ae10f09469fa4f913e049002481
I20250112 20:51:45.141793 983 leader_election.cc:304] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.141731 1164 raft_consensus.cc:383] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:45.141988 983 raft_consensus.cc:2681] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:45.142445 2447 ts_tablet_manager.cc:1905] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.143245 1164 raft_consensus.cc:738] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.143522 1164 consensus_queue.cc:260] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.144337 1178 consensus_queue.cc:260] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.145221 1163 consensus_queue.cc:260] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.146692 2521 raft_consensus.cc:491] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.146811 2521 raft_consensus.cc:513] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.147395 2521 leader_election.cc:290] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.148236 2447 ts_tablet_manager.cc:1918] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.148322 2447 log.cc:1198] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/54ab0ae10f09469fa4f913e049002481
I20250112 20:51:45.148599 2447 ts_tablet_manager.cc:1939] T 54ab0ae10f09469fa4f913e049002481 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.149283 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.149814 847 leader_election.cc:343] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.152288 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7b12d34248404d5485e174ce460e7d56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.153085 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b12d34248404d5485e174ce460e7d56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.156042 1178 ts_tablet_manager.cc:1428] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250112 20:51:45.161460 1164 ts_tablet_manager.cc:1428] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.058s user 0.001s sys 0.002s
I20250112 20:51:45.162313 1164 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.167698 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_22f40a2d9f084970b9d2152f401edd49" } new_table_name: "default.Table_2cc701502bdb4a88ab61b7f72cb1ab77" modify_external_catalogs: true
I20250112 20:51:45.169137 1163 ts_tablet_manager.cc:1428] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250112 20:51:45.163683 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.170519 2208 consensus_queue.cc:1035] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.175172 1164 tablet_bootstrap.cc:654] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:45.176700 847 leader_election.cc:343] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.176851 847 leader_election.cc:304] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.177229 2108 raft_consensus.cc:2743] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.184520 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 54ab0ae10f09469fa4f913e049002481 (table default.Table_a374fefd5800476d8e57e23cf0577a2a [id=54df980379dc4ba7b4683f7c484204c5]) successfully deleted
W20250112 20:51:45.189774 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e584b761d234c5aa0ab875acd8e1e05 consensus state has no leader
I20250112 20:51:45.191298 2108 raft_consensus.cc:491] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.191454 2108 raft_consensus.cc:513] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.192041 2108 leader_election.cc:290] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:45.192993 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.193697 847 leader_election.cc:343] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.193956 2447 tablet_replica.cc:331] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.194154 2447 raft_consensus.cc:2238] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.194312 2447 raft_consensus.cc:2267] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.194747 1178 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.194809 2447 ts_tablet_manager.cc:1905] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.195343 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:45.196043 847 leader_election.cc:343] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.196166 847 leader_election.cc:304] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.196552 1178 tablet_bootstrap.cc:654] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.196606 2108 raft_consensus.cc:2743] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.204841 1163 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.206804 1163 tablet_bootstrap.cc:654] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.207167 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2ac80cd3fa134256b29e1eb39796aa6d" } new_table_name: "default.Table_bd0bb2087a8b4226ac58760eb0cc949d" modify_external_catalogs: true
I20250112 20:51:45.210711 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.213366 2108 raft_consensus.cc:491] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.213425 2447 ts_tablet_manager.cc:1918] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.213507 2447 log.cc:1198] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/bc05d287d81c46558acdd9a42edda6f8
I20250112 20:51:45.213528 2108 raft_consensus.cc:513] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.213783 2447 ts_tablet_manager.cc:1939] T bc05d287d81c46558acdd9a42edda6f8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.214133 2108 leader_election.cc:290] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.214552 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.215132 847 leader_election.cc:343] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.228271 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:45.231374 847 leader_election.cc:343] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.231518 847 leader_election.cc:304] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.231997 2648 raft_consensus.cc:2743] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.232586 1164 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:45.240628 1042 raft_consensus.cc:3054] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.242648 2447 tablet_replica.cc:331] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.242863 2447 raft_consensus.cc:2238] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.243041 2447 raft_consensus.cc:2267] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.243556 2447 ts_tablet_manager.cc:1905] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.244925 2108 raft_consensus.cc:491] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.245043 2108 raft_consensus.cc:513] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.245600 2108 leader_election.cc:290] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:45.246496 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.247018 2108 raft_consensus.cc:491] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.247153 2108 raft_consensus.cc:513] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.247788 2108 leader_election.cc:290] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:45.248561 847 leader_election.cc:343] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.249065 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.250622 847 leader_election.cc:343] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.254410 1049 raft_consensus.cc:3054] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.257954 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.263438 1164 ts_tablet_manager.cc:1397] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.002s
I20250112 20:51:45.258177 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.258709 2059 consensus_queue.cc:1035] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.263904 1178 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.264027 1178 ts_tablet_manager.cc:1397] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20250112 20:51:45.253558 1185 ts_tablet_manager.cc:1918] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:45.265214 1185 log.cc:1198] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/30156185f6104eada87b936fc1829df9
I20250112 20:51:45.265641 1185 ts_tablet_manager.cc:1939] T 30156185f6104eada87b936fc1829df9 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:45.266844 1185 tablet_replica.cc:331] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:45.267041 1185 raft_consensus.cc:2238] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:45.267246 1185 pending_rounds.cc:62] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:45.267299 1185 pending_rounds.cc:69] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585067884617728 op_type: NO_OP noop_request { }
I20250112 20:51:45.267395 1185 raft_consensus.cc:2883] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:45.267475 1185 raft_consensus.cc:2267] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:45.267928 847 leader_election.cc:343] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.268043 847 leader_election.cc:304] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.268481 2108 raft_consensus.cc:2743] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:45.269052 847 leader_election.cc:343] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.269176 847 leader_election.cc:304] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.269582 2108 raft_consensus.cc:2743] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:45.272747 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efe90c40f96c4736954e22a127ced1b7 consensus state has no leader
I20250112 20:51:45.274912 2108 raft_consensus.cc:491] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.275079 2108 raft_consensus.cc:513] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.275815 2108 leader_election.cc:290] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.282627 2613 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250112 20:51:45.289849 2447 ts_tablet_manager.cc:1918] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:45.289968 2447 log.cc:1198] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/30156185f6104eada87b936fc1829df9
I20250112 20:51:45.291508 2447 ts_tablet_manager.cc:1939] T 30156185f6104eada87b936fc1829df9 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.264824 1164 raft_consensus.cc:357] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.296957 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.298138 2447 tablet_replica.cc:331] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.298336 2447 raft_consensus.cc:2238] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.298489 2447 raft_consensus.cc:2267] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:45.298660 847 leader_election.cc:343] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.298995 2751 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:45.300009 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.300196 776 raft_consensus.cc:2463] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:45.298044 1163 tablet_bootstrap.cc:492] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.301759 1163 ts_tablet_manager.cc:1397] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20250112 20:51:45.295780 1164 raft_consensus.cc:383] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.302145 1164 raft_consensus.cc:738] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.302386 1164 consensus_queue.cc:260] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.303117 1164 ts_tablet_manager.cc:1428] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250112 20:51:45.303580 2447 ts_tablet_manager.cc:1905] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.307478 1178 raft_consensus.cc:357] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.307672 1178 raft_consensus.cc:383] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.307734 1178 raft_consensus.cc:738] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.308621 1178 consensus_queue.cc:260] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.309849 1178 ts_tablet_manager.cc:1428] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20250112 20:51:45.311168 2059 raft_consensus.cc:491] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.311285 2059 raft_consensus.cc:513] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.311805 2059 leader_election.cc:290] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:45.313041 2648 raft_consensus.cc:491] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.313160 2648 raft_consensus.cc:513] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.313660 2648 leader_election.cc:290] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:45.314522 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:45.315755 2747 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250112 20:51:45.315760 2613 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:45.318768 1163 raft_consensus.cc:357] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.318946 1163 raft_consensus.cc:383] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.319000 1163 raft_consensus.cc:738] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.319208 1163 consensus_queue.cc:260] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.319783 1163 ts_tablet_manager.cc:1428] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 20:51:45.320034 1163 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.320442 2785 raft_consensus.cc:491] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.320544 2785 raft_consensus.cc:513] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.321161 2785 leader_election.cc:290] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.322324 2563 raft_consensus.cc:491] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.322759 2563 raft_consensus.cc:513] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.323320 2563 leader_election.cc:290] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:45.324007 1178 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.327111 1163 tablet_bootstrap.cc:654] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.328040 1178 tablet_bootstrap.cc:654] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:45.332213 983 leader_election.cc:343] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 903027fe49e34a308108356ba89a9c3a
I20250112 20:51:45.240480 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet bc05d287d81c46558acdd9a42edda6f8 (table default.Table_6caf53d8232c4928a6a597866d27c8dd [id=de2fbf3d0fcd4c108981cd0ca4c87a92]) successfully deleted
W20250112 20:51:45.332965 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 92506.7us
W20250112 20:51:45.333343 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet eaa091fe2d3c4f72815f228ac3e39128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaa091fe2d3c4f72815f228ac3e39128 already in progress: creating tablet
W20250112 20:51:45.333596 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3b299e16586145ee9ed5a7b118c049b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b299e16586145ee9ed5a7b118c049b3 already in progress: creating tablet
I20250112 20:51:45.333873 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]) successfully deleted
W20250112 20:51:45.334133 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8098069639a2435ab500908db9ecd12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8098069639a2435ab500908db9ecd12a already in progress: creating tablet
W20250112 20:51:45.334332 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 already in progress: creating tablet
I20250112 20:51:45.334834 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]) successfully deleted
W20250112 20:51:45.335088 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8098069639a2435ab500908db9ecd12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8098069639a2435ab500908db9ecd12a already in progress: creating tablet
W20250112 20:51:45.335361 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 already in progress: creating tablet
I20250112 20:51:45.338382 2751 tablet.cc:1721] T 24c8dc85ac9240598d6e413deac19abe P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250112 20:51:45.339229 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:45.338521 710 consensus_peers.cc:487] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.339450 908 raft_consensus.cc:2463] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:45.340555 2747 tablet.cc:1721] T 55d75f9b410f4790ac926646a3cf0fd0 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:45.341293 847 leader_election.cc:304] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.341612 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.341742 2648 raft_consensus.cc:2798] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.341811 2648 raft_consensus.cc:491] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.341871 2648 raft_consensus.cc:3054] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.341856 1042 raft_consensus.cc:2463] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:45.342559 2447 ts_tablet_manager.cc:1918] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:45.342648 2447 log.cc:1198] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:45.343108 2447 ts_tablet_manager.cc:1939] T 6092a31de9bc47e4bae5f5aebb180c6d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.344269 2648 raft_consensus.cc:513] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.344484 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 6092a31de9bc47e4bae5f5aebb180c6d (table default.Table_4fb0c297ce434ae4be2c6eba23a061a6 [id=8e9b87e308d14762bfeb115e867b17c9]) successfully deleted
I20250112 20:51:45.345901 2521 consensus_queue.cc:1035] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:45.346506 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:45.346738 847 leader_election.cc:304] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.347028 2521 raft_consensus.cc:2743] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.344792 2648 leader_election.cc:290] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.348884 1164 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:45.348639 983 leader_election.cc:343] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9856e6b625d04f17b0869cee08f69142
W20250112 20:51:45.349588 983 leader_election.cc:343] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 9856e6b625d04f17b0869cee08f69142
I20250112 20:51:45.349715 2447 tablet_replica.cc:331] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.349932 2447 raft_consensus.cc:2238] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.350085 2447 raft_consensus.cc:2267] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.350550 2447 ts_tablet_manager.cc:1905] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.350625 1164 tablet_bootstrap.cc:654] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.352123 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_770aef6ded7a457fb8248f61e8c3a65c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.349730 983 leader_election.cc:304] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.353195 2787 raft_consensus.cc:2743] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.353091 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ff262f32148c4bb9acd0b6492fe0b8e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.353461 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:45.353566 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff262f32148c4bb9acd0b6492fe0b8e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:45.353977 847 leader_election.cc:343] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.355104 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_22606a15a61946b08eec7776233747ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.355499 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22606a15a61946b08eec7776233747ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:45.352738 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_770aef6ded7a457fb8248f61e8c3a65c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.357854 1163 tablet_bootstrap.cc:492] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.357990 1163 ts_tablet_manager.cc:1397] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
W20250112 20:51:45.358395 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da93a5d375334552bc68510aebb507dd consensus state has no leader
I20250112 20:51:45.361094 1163 raft_consensus.cc:357] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.361335 1163 raft_consensus.cc:383] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.362577 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_42c5efdd34e643cba03f44dc9c276e53" } modify_external_catalogs: true
I20250112 20:51:45.361387 1163 raft_consensus.cc:738] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.362856 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_42c5efdd34e643cba03f44dc9c276e53" } modify_external_catalogs: true
I20250112 20:51:45.362946 1163 consensus_queue.cc:260] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.363477 1163 ts_tablet_manager.cc:1428] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:45.368520 1178 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.368654 1178 ts_tablet_manager.cc:1397] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250112 20:51:45.369571 2787 raft_consensus.cc:491] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.369686 2787 raft_consensus.cc:513] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.370299 2787 leader_election.cc:290] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:45.371156 983 leader_election.cc:343] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:45.369341 1178 raft_consensus.cc:357] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.375366 1178 raft_consensus.cc:383] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.375443 1178 raft_consensus.cc:738] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.375660 1178 consensus_queue.cc:260] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.376286 1178 ts_tablet_manager.cc:1428] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250112 20:51:45.376155 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.376530 1178 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:45.376662 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.379165 1178 tablet_bootstrap.cc:654] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.379361 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 015f54bdae6b4333963f9e085c5189e8 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.380122 1185 ts_tablet_manager.cc:1905] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.380668 1163 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.382514 1163 tablet_bootstrap.cc:654] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.387902 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:45.397677 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:45.397933 454 catalog_manager.cc:5526] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:45.398387 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:45.398550 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:45.398576 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 575f873137ad4b85b5f957fa8665c416 (DEFAULT_TABLE table=default.Table_35d70052f03445b3be247723d56eec3d [id=4c26f80ad9f04db1a6f2e2beb5f7df04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.398958 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f873137ad4b85b5f957fa8665c416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.398798 454 catalog_manager.cc:5526] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:45.399259 753 tablet_service.cc:1467] Processing CreateTablet for tablet 575f873137ad4b85b5f957fa8665c416 (DEFAULT_TABLE table=default.Table_35d70052f03445b3be247723d56eec3d [id=4c26f80ad9f04db1a6f2e2beb5f7df04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.399369 454 catalog_manager.cc:5526] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:45.399613 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f873137ad4b85b5f957fa8665c416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.399749 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:45.399964 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:45.400256 710 leader_election.cc:304] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:45.400622 2750 raft_consensus.cc:2798] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.400693 2750 raft_consensus.cc:491] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.400682 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 89f9b46da06a476ab9f2a93ca74ff736
I20250112 20:51:45.400750 2750 raft_consensus.cc:3054] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.401257 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 886690f24fa540a6b496c926bc7a2b07 (DEFAULT_TABLE table=default.Table_90428f7ce05e4667a57dc6fec766a9be [id=084e44b53d4e48b3a0cb8cd4d1aca3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.401315 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90bbde70dc014c87be25b842e16d44d0" } modify_external_catalogs: true
I20250112 20:51:45.401463 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90bbde70dc014c87be25b842e16d44d0" } modify_external_catalogs: true
I20250112 20:51:45.401582 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886690f24fa540a6b496c926bc7a2b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.403046 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.404265 2750 raft_consensus.cc:513] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.404841 2750 leader_election.cc:290] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:45.405068 983 leader_election.cc:343] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 903027fe49e34a308108356ba89a9c3a
I20250112 20:51:45.405175 983 leader_election.cc:304] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.405761 885 tablet_service.cc:1467] Processing CreateTablet for tablet 575f873137ad4b85b5f957fa8665c416 (DEFAULT_TABLE table=default.Table_35d70052f03445b3be247723d56eec3d [id=4c26f80ad9f04db1a6f2e2beb5f7df04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.406105 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f873137ad4b85b5f957fa8665c416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.408916 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.409369 2789 raft_consensus.cc:2743] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.410593 2563 raft_consensus.cc:491] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.410717 2563 raft_consensus.cc:513] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.411195 2563 leader_election.cc:290] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.411875 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f9b46da06a476ab9f2a93ca74ff736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.411991 2648 raft_consensus.cc:491] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.412127 2648 raft_consensus.cc:513] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.412302 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:45.412508 1049 raft_consensus.cc:3054] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.412678 2648 leader_election.cc:290] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.412977 1042 raft_consensus.cc:3054] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.415467 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5d0474b9315542a68d09ba65bf803253"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.416033 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d0474b9315542a68d09ba65bf803253 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:45.417246 983 leader_election.cc:343] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 4561956c97824824a55283a8da21929e
I20250112 20:51:45.415056 1049 raft_consensus.cc:2463] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:45.417457 2648 consensus_queue.cc:1035] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.418046 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9178cf782d4bd28eba90cc03796885" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:45.418193 912 raft_consensus.cc:3054] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.420580 912 raft_consensus.cc:2463] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:45.420951 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.421394 847 leader_election.cc:343] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.421452 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 015f54bdae6b4333963f9e085c5189e8
I20250112 20:51:45.421917 449 catalog_manager.cc:6596] ReplaceTablet: tablet 015f54bdae6b4333963f9e085c5189e8 deleted and replaced by tablet 1b228712feb24dd1af8d8f52be35b8c8
I20250112 20:51:45.422235 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cad323ed70af4a0a91136d819296a4da" } modify_external_catalogs: true
I20250112 20:51:45.422379 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cad323ed70af4a0a91136d819296a4da" } modify_external_catalogs: true
I20250112 20:51:45.424595 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 015f54bdae6b4333963f9e085c5189e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b228712feb24dd1af8d8f52be35b8c8) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.424901 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 015f54bdae6b4333963f9e085c5189e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b228712feb24dd1af8d8f52be35b8c8) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.425364 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:45.425527 773 raft_consensus.cc:3054] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.426549 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 015f54bdae6b4333963f9e085c5189e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b228712feb24dd1af8d8f52be35b8c8) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.427426 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0d072ebae78e40c48e158eb65775476c
I20250112 20:51:45.427923 448 catalog_manager.cc:6596] ReplaceTablet: tablet 0d072ebae78e40c48e158eb65775476c deleted and replaced by tablet 6f81ce5122864ba79e643d6de8b6f857
I20250112 20:51:45.428393 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f587cdf889844d2892126c8d93dc87be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.428897 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f587cdf889844d2892126c8d93dc87be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.432911 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d072ebae78e40c48e158eb65775476c with delete_type TABLET_DATA_DELETED (replaced by tablet 6f81ce5122864ba79e643d6de8b6f857) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.433175 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d072ebae78e40c48e158eb65775476c with delete_type TABLET_DATA_DELETED (replaced by tablet 6f81ce5122864ba79e643d6de8b6f857) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.434455 773 raft_consensus.cc:2463] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:45.435600 710 consensus_peers.cc:487] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.435678 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.436048 710 leader_election.cc:304] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:45.436321 2748 raft_consensus.cc:2798] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.436432 2748 raft_consensus.cc:695] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:45.436651 2748 consensus_queue.cc:237] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.437264 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d072ebae78e40c48e158eb65775476c with delete_type TABLET_DATA_DELETED (replaced by tablet 6f81ce5122864ba79e643d6de8b6f857) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.438089 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_b0f906299bab48059b43f99eee14bc49 [id=1ef2730aa400453a9d078c520a5329f8] alter complete (version 2)
I20250112 20:51:45.440197 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3f6ad2ac50c745448031bdf94e0eea59
I20250112 20:51:45.440647 455 catalog_manager.cc:6596] ReplaceTablet: tablet 3f6ad2ac50c745448031bdf94e0eea59 deleted and replaced by tablet 0d14b0bfd973497b97ed6a3c7bec1aa6
I20250112 20:51:45.441125 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_03840c2c47d242e6b6459a38152c5afc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.441268 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d14b0bfd973497b97ed6a3c7bec1aa6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.441460 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d14b0bfd973497b97ed6a3c7bec1aa6) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.441540 1164 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:45.441627 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03840c2c47d242e6b6459a38152c5afc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.441669 1164 ts_tablet_manager.cc:1397] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.001s
I20250112 20:51:45.442710 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ad2ac50c745448031bdf94e0eea59 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d14b0bfd973497b97ed6a3c7bec1aa6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.442651 1164 raft_consensus.cc:357] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.442934 1164 raft_consensus.cc:383] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.443009 1164 raft_consensus.cc:738] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.443240 1164 consensus_queue.cc:260] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.443831 1164 ts_tablet_manager.cc:1428] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:45.449883 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 69012f97aac1441095aa3bb1548de9cd
I20250112 20:51:45.451418 2447 ts_tablet_manager.cc:1918] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.451587 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 69012f97aac1441095aa3bb1548de9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.452174 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_29c0f892c9954a598f7f27c4be906a91" } new_table_name: "default.Table_f2b30f50ce0843e2861e9bae518ea27b" modify_external_catalogs: true
I20250112 20:51:45.452243 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 69012f97aac1441095aa3bb1548de9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.452454 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 69012f97aac1441095aa3bb1548de9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.456899 2108 raft_consensus.cc:491] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.457020 2108 raft_consensus.cc:513] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.457604 2108 leader_election.cc:290] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.457998 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc22207e9204b73877c74f61ebba8a8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.458535 847 leader_election.cc:343] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.459666 753 tablet_service.cc:1467] Processing CreateTablet for tablet 886690f24fa540a6b496c926bc7a2b07 (DEFAULT_TABLE table=default.Table_90428f7ce05e4667a57dc6fec766a9be [id=084e44b53d4e48b3a0cb8cd4d1aca3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.460016 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886690f24fa540a6b496c926bc7a2b07. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:45.460282 983 leader_election.cc:343] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:45.460305 885 tablet_service.cc:1467] Processing CreateTablet for tablet 886690f24fa540a6b496c926bc7a2b07 (DEFAULT_TABLE table=default.Table_90428f7ce05e4667a57dc6fec766a9be [id=084e44b53d4e48b3a0cb8cd4d1aca3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.460386 983 leader_election.cc:304] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.460644 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886690f24fa540a6b496c926bc7a2b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.460755 2563 raft_consensus.cc:2743] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:45.461417 983 leader_election.cc:343] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 4561956c97824824a55283a8da21929e
I20250112 20:51:45.461509 983 leader_election.cc:304] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.461822 2563 raft_consensus.cc:2743] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.462704 847 leader_election.cc:304] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.462996 2108 raft_consensus.cc:2798] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.463097 2108 raft_consensus.cc:695] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:45.463301 2108 consensus_queue.cc:237] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:45.466344 847 leader_election.cc:343] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.466441 847 leader_election.cc:304] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.466742 2648 raft_consensus.cc:2743] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.468079 2447 log.cc:1198] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:45.468392 2447 ts_tablet_manager.cc:1939] T 2ab635fec8a64efd8e0c527d913ccd7d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.469187 2447 tablet_replica.cc:331] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.469362 2447 raft_consensus.cc:2238] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.469475 2447 raft_consensus.cc:2267] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.469947 2447 ts_tablet_manager.cc:1905] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.470077 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 2ab635fec8a64efd8e0c527d913ccd7d (table default.Table_f54731064e4948fdbca8cf87ae2a9533 [id=45ea3bd35f9141e9ae62c26aa0138259]) successfully deleted
I20250112 20:51:45.474654 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c2664a2e98f74f73ab2dc0adaa2335d3
I20250112 20:51:45.475860 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.477172 875 tablet_service.cc:1514] Processing DeleteTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.477381 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.477949 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f9c81d06d5cb4e3cbc3367527dcb10d3" } new_table_name: "default.Table_1c9cc0e4c5ea4333a8e2fcc16ff4ac0f" modify_external_catalogs: true
I20250112 20:51:45.481030 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:45.481722 450 master_service.cc:946] ReplaceTablet: received request to replace tablet fc1335522a384eb98163a2539630b93e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.482851 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b0786594542416cb67c4112921943b3
I20250112 20:51:45.483597 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_64d69b6c7750480191ca2e8f3234b1e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.484086 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d69b6c7750480191ca2e8f3234b1e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.485466 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0786594542416cb67c4112921943b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.487124 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.487197 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0786594542416cb67c4112921943b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.487412 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.488220 1164 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.489662 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ef05483d74ba466ea37bf2e84489fc15" } new_table_name: "default.Table_4f98a4c818a2481d8f8e64ea64693585" modify_external_catalogs: true
I20250112 20:51:45.489931 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:45.490223 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:45.490466 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:45.493561 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0786594542416cb67c4112921943b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.493961 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56b47cebed483fa9679158c9cae314 (DEFAULT_TABLE table=default.Table_23ec53724daa45fa96c695571826a6ea [id=bfc5d874840f4275b15de6e2bd3a1a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.494325 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f56b47cebed483fa9679158c9cae314. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.494971 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56b47cebed483fa9679158c9cae314 (DEFAULT_TABLE table=default.Table_23ec53724daa45fa96c695571826a6ea [id=bfc5d874840f4275b15de6e2bd3a1a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.497670 875 tablet_service.cc:1467] Processing CreateTablet for tablet 2f56b47cebed483fa9679158c9cae314 (DEFAULT_TABLE table=default.Table_23ec53724daa45fa96c695571826a6ea [id=bfc5d874840f4275b15de6e2bd3a1a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.498098 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f56b47cebed483fa9679158c9cae314. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:45.500393 847 consensus_peers.cc:487] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30156185f6104eada87b936fc1829df9. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.504945 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 01367a11dea04b33a9255508501b9060
I20250112 20:51:45.505486 449 master_service.cc:946] ReplaceTablet: received request to replace tablet d80791d8582446b89adf212d3d6507eb from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.506412 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 01367a11dea04b33a9255508501b9060 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.506814 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc22207e9204b73877c74f61ebba8a8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.507053 773 raft_consensus.cc:2463] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:45.508020 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d571de1e2b104ba480cdaf2523b54599 (DEFAULT_TABLE table=default.Table_a00f71eb25f84488a78ed3bea61f57e7 [id=b13df04162c64cb9bc83f1b88b6d6503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.508385 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d571de1e2b104ba480cdaf2523b54599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.510517 1185 ts_tablet_manager.cc:1918] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:45.510618 1185 log.cc:1198] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:45.510947 1185 ts_tablet_manager.cc:1939] T 2e6b4882a3d744bda5677aa9e01a53f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:45.511538 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 01367a11dea04b33a9255508501b9060 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.511745 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 01367a11dea04b33a9255508501b9060 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.513334 847 leader_election.cc:304] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.513805 2521 raft_consensus.cc:2798] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.513906 2521 raft_consensus.cc:491] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.513980 2521 raft_consensus.cc:3054] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.513602 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_316b86d7610444219e7d6f52c7d19556 [id=3a4de1859ffd4db3931ce68febdd8836] alter complete (version 5)
I20250112 20:51:45.517787 2521 raft_consensus.cc:513] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.518435 2521 leader_election.cc:290] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.519917 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc22207e9204b73877c74f61ebba8a8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:45.520305 847 leader_election.cc:343] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.521787 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f669a739d94f99a5b7e9ccccd29545 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:45.524188 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b930069b6f1b49439dbc0a7abd3fba72 consensus state has no leader
I20250112 20:51:45.525108 2108 raft_consensus.cc:491] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.525218 2108 raft_consensus.cc:513] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.526381 2108 leader_election.cc:290] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.526428 875 tablet_service.cc:1467] Processing CreateTablet for tablet d571de1e2b104ba480cdaf2523b54599 (DEFAULT_TABLE table=default.Table_a00f71eb25f84488a78ed3bea61f57e7 [id=b13df04162c64cb9bc83f1b88b6d6503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.526620 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.526917 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d571de1e2b104ba480cdaf2523b54599. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:45.528369 847 leader_election.cc:343] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.529374 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 144570330d8d47e1b4f56cf127266232 (DEFAULT_TABLE table=default.Table_13c00ee8163d4ea0b5bd8ed71ff8a80c [id=26de745a311d4840b161639072373122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.529670 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144570330d8d47e1b4f56cf127266232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.533145 2563 raft_consensus.cc:491] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.533262 2563 raft_consensus.cc:513] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.533775 2563 leader_election.cc:290] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:45.534657 983 leader_election.cc:343] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d706f88f14ab4fd9a459944863afaacd
I20250112 20:51:45.534687 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:45.534977 2632 consensus_queue.cc:1035] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.536413 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_84c10c7cd6164c07aed6ccf21fe30476" } new_table_name: "default.Table_036c5b0e81ef4d9ea5b52c8ca64a67e9" modify_external_catalogs: true
I20250112 20:51:45.537988 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eab8a830ea0c47409a6ae483e247cbe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.538461 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eab8a830ea0c47409a6ae483e247cbe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.540038 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_201e5a51990a4629a810d8e3d9226ce7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.540591 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_201e5a51990a4629a810d8e3d9226ce7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.541601 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc22207e9204b73877c74f61ebba8a8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:45.541767 774 raft_consensus.cc:3054] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.542181 1178 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.542313 1178 ts_tablet_manager.cc:1397] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.166s user 0.008s sys 0.010s
I20250112 20:51:45.542953 1178 raft_consensus.cc:357] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.543087 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.543174 1178 raft_consensus.cc:383] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.543236 1178 raft_consensus.cc:738] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.543486 1178 consensus_queue.cc:260] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.544389 1178 ts_tablet_manager.cc:1428] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:45.544451 2447 ts_tablet_manager.cc:1918] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.544562 2447 log.cc:1198] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/903027fe49e34a308108356ba89a9c3a
I20250112 20:51:45.544646 1178 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.544862 2447 ts_tablet_manager.cc:1939] T 903027fe49e34a308108356ba89a9c3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.546032 1178 tablet_bootstrap.cc:654] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.548043 1163 tablet_bootstrap.cc:492] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.548187 1163 ts_tablet_manager.cc:1397] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.168s user 0.007s sys 0.010s
W20250112 20:51:45.548482 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 consensus state has no leader
I20250112 20:51:45.543252 776 raft_consensus.cc:2463] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:45.549038 1163 raft_consensus.cc:357] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.549240 1163 raft_consensus.cc:383] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.549295 1163 raft_consensus.cc:738] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.549293 753 tablet_service.cc:1467] Processing CreateTablet for tablet d571de1e2b104ba480cdaf2523b54599 (DEFAULT_TABLE table=default.Table_a00f71eb25f84488a78ed3bea61f57e7 [id=b13df04162c64cb9bc83f1b88b6d6503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.549499 1163 consensus_queue.cc:260] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.549662 737 tablet_service.cc:1467] Processing CreateTablet for tablet 144570330d8d47e1b4f56cf127266232 (DEFAULT_TABLE table=default.Table_13c00ee8163d4ea0b5bd8ed71ff8a80c [id=26de745a311d4840b161639072373122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.553939 774 raft_consensus.cc:2463] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:45.554888 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f56b47cebed483fa9679158c9cae314. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.549986 1163 ts_tablet_manager.cc:1428] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:45.559389 1163 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.560714 1163 tablet_bootstrap.cc:654] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.561342 447 catalog_manager.cc:5526] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:45.561877 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
W20250112 20:51:45.562060 983 leader_election.cc:343] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d706f88f14ab4fd9a459944863afaacd
I20250112 20:51:45.562165 983 leader_election.cc:304] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.562286 847 leader_election.cc:304] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.562160 447 catalog_manager.cc:5526] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:45.562500 2648 raft_consensus.cc:2798] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.562556 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:45.562592 2648 raft_consensus.cc:491] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.562654 2648 raft_consensus.cc:3054] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.562633 2563 raft_consensus.cc:2743] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.562759 847 leader_election.cc:304] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.563194 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet b930069b6f1b49439dbc0a7abd3fba72 (table default.Table_f2b30f50ce0843e2861e9bae518ea27b [id=61af9ec286544b919c9f053b647bc6fe]). Expected version 1 got 0
I20250112 20:51:45.563336 2521 raft_consensus.cc:2798] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.565446 2648 raft_consensus.cc:513] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.565708 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.565878 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet fc1335522a384eb98163a2539630b93e
I20250112 20:51:45.566367 450 catalog_manager.cc:6596] ReplaceTablet: tablet fc1335522a384eb98163a2539630b93e deleted and replaced by tablet e65c89f6026747dd88f960a638087607
I20250112 20:51:45.566397 2648 leader_election.cc:290] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.567112 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1335522a384eb98163a2539630b93e with delete_type TABLET_DATA_DELETED (replaced by tablet e65c89f6026747dd88f960a638087607) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.569032 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_093ad5dba68a4f0b9d96c37ac3888a83" } new_table_name: "default.Table_3ec2de0c20494221b62ee138a5073ca3" modify_external_catalogs: true
I20250112 20:51:45.569921 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ca466d6d99ca472db8704c08e6b6b4d2 (DEFAULT_TABLE table=default.Table_f1cd4dd5cbfb498b9337d35cac1bd249 [id=0e95338aed1148468d473290200b46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.570097 754 tablet_service.cc:1467] Processing CreateTablet for tablet ca466d6d99ca472db8704c08e6b6b4d2 (DEFAULT_TABLE table=default.Table_f1cd4dd5cbfb498b9337d35cac1bd249 [id=0e95338aed1148468d473290200b46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.570283 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca466d6d99ca472db8704c08e6b6b4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.570421 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca466d6d99ca472db8704c08e6b6b4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.572546 875 tablet_service.cc:1467] Processing CreateTablet for tablet 144570330d8d47e1b4f56cf127266232 (DEFAULT_TABLE table=default.Table_13c00ee8163d4ea0b5bd8ed71ff8a80c [id=26de745a311d4840b161639072373122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.572932 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144570330d8d47e1b4f56cf127266232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.574710 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_96d7128d31fe403abd7f96408718800c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.575170 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96d7128d31fe403abd7f96408718800c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.576392 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1335522a384eb98163a2539630b93e with delete_type TABLET_DATA_DELETED (replaced by tablet e65c89f6026747dd88f960a638087607) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.577279 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1335522a384eb98163a2539630b93e with delete_type TABLET_DATA_DELETED (replaced by tablet e65c89f6026747dd88f960a638087607) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.578696 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d9a46ff1902a48529ee58d8c36e0d9e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.578758 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:45.579155 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9a46ff1902a48529ee58d8c36e0d9e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.580981 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d17a39683ebe4d8c9b49bf0de58ed4bd (DEFAULT_TABLE table=default.Table_d1817a082b7f4497b916c47271738874 [id=97b4016a813447a9ad1b07a189d18fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.581367 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a39683ebe4d8c9b49bf0de58ed4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.583323 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:45.583514 774 raft_consensus.cc:3054] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.585996 774 raft_consensus.cc:2463] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:45.586014 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_826881febc214de081501f6ecb406972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.586506 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_826881febc214de081501f6ecb406972 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.590312 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.590652 741 tablet_service.cc:1467] Processing CreateTablet for tablet d17a39683ebe4d8c9b49bf0de58ed4bd (DEFAULT_TABLE table=default.Table_d1817a082b7f4497b916c47271738874 [id=97b4016a813447a9ad1b07a189d18fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.590662 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea9425f3e7b46288c81dc8cdade99b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.573029 885 tablet_service.cc:1467] Processing CreateTablet for tablet ca466d6d99ca472db8704c08e6b6b4d2 (DEFAULT_TABLE table=default.Table_f1cd4dd5cbfb498b9337d35cac1bd249 [id=0e95338aed1148468d473290200b46ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:45.579450 847 leader_election.cc:343] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:45.592628 847 consensus_peers.cc:487] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30156185f6104eada87b936fc1829df9. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.592829 847 leader_election.cc:304] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.592890 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca466d6d99ca472db8704c08e6b6b4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.593087 2648 raft_consensus.cc:2798] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.593197 2648 raft_consensus.cc:695] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:45.593401 2648 consensus_queue.cc:237] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.593644 881 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.593961 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea9425f3e7b46288c81dc8cdade99b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.597102 884 tablet_service.cc:1467] Processing CreateTablet for tablet d17a39683ebe4d8c9b49bf0de58ed4bd (DEFAULT_TABLE table=default.Table_d1817a082b7f4497b916c47271738874 [id=97b4016a813447a9ad1b07a189d18fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.597477 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a39683ebe4d8c9b49bf0de58ed4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.600354 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 (DEFAULT_TABLE table=default.Table_a6b14949b43a4bbd81fd8004f55380bb [id=cbdd1fab16ae484bb212104015c6e747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.600703 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea9425f3e7b46288c81dc8cdade99b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.600878 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d17a39683ebe4d8c9b49bf0de58ed4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.604241 2632 consensus_queue.cc:1035] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.605672 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6052e51a7476a955e7d0f8a9977fd (DEFAULT_TABLE table=default.Table_1f284beef9d64af2a22525800952a18b [id=1e9b52f662aa489eb9616cc8ad3527be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.605976 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6052e51a7476a955e7d0f8a9977fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.608687 1164 tablet_bootstrap.cc:654] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.609117 2563 raft_consensus.cc:491] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.609246 2563 raft_consensus.cc:513] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.609800 2563 leader_election.cc:290] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:45.614017 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 903027fe49e34a308108356ba89a9c3a (table default.Table_c0e48e09ccab48a9a2cf6aa4212f1658 [id=dd6576318c1d4470ab4f7facffe21912]) successfully deleted
I20250112 20:51:45.614662 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:45.615058 912 raft_consensus.cc:2388] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
W20250112 20:51:45.615561 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 549d92191a594788bf639cd60a6b5142 consensus state has no leader
I20250112 20:51:45.619537 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
I20250112 20:51:45.620452 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc09e8e983ec49c2b224a4b5273bacc5" } modify_external_catalogs: true
I20250112 20:51:45.620628 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bc09e8e983ec49c2b224a4b5273bacc5" } modify_external_catalogs: true
W20250112 20:51:45.622609 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3dc719cd84124082b782016011e37c8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dc719cd84124082b782016011e37c8b already in progress: creating tablet
W20250112 20:51:45.623111 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 942e8f74c4334c0eb4b441c020605631 consensus state has no leader
I20250112 20:51:45.623260 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b5c457468d7d49078cb67c543721f334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.623731 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5c457468d7d49078cb67c543721f334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:45.625869 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cc9d5d182827465abd3799d3ad9a6484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9d5d182827465abd3799d3ad9a6484 already in progress: creating tablet
W20250112 20:51:45.628118 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d80791d8582446b89adf212d3d6507eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d80791d8582446b89adf212d3d6507eb already in progress: creating tablet
I20250112 20:51:45.628430 2447 tablet_replica.cc:331] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.628670 2447 raft_consensus.cc:2238] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.628913 2447 raft_consensus.cc:2267] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.628815 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d477fb5e5ad5447bb182bc9998561a43" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.639792 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d571de1e2b104ba480cdaf2523b54599. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.640105 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 2e6b4882a3d744bda5677aa9e01a53f4 (table default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]) successfully deleted
I20250112 20:51:45.641803 741 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6052e51a7476a955e7d0f8a9977fd (DEFAULT_TABLE table=default.Table_1f284beef9d64af2a22525800952a18b [id=1e9b52f662aa489eb9616cc8ad3527be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.642119 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6052e51a7476a955e7d0f8a9977fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.642911 1185 tablet_replica.cc:331] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:45.643115 1185 raft_consensus.cc:2238] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:45.587221 1186 ts_tablet_manager.cc:1939] T 3343e1d7f1d04a498bd0ee00311b6a61 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:45.643505 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 854) took 1841 ms. Trace:
I20250112 20:51:45.643587 1186 rpcz_store.cc:276] 0112 20:51:43.802104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.823354 (+ 21250us) service_pool.cc:225] Handling call
0112 20:51:45.022777 (+1199423us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:45.022788 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.050043 (+ 27255us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.136875 (+ 86832us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.643491 (+506616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199230}
I20250112 20:51:45.643976 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3343e1d7f1d04a498bd0ee00311b6a61 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
I20250112 20:51:45.644347 1185 raft_consensus.cc:2267] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.645206 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144570330d8d47e1b4f56cf127266232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.645385 1185 ts_tablet_manager.cc:1905] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.653214 2447 ts_tablet_manager.cc:1905] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.654301 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d3a871e1d7194bfdb60a286d2fe92c1c" } modify_external_catalogs: true
I20250112 20:51:45.654523 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d3a871e1d7194bfdb60a286d2fe92c1c" } modify_external_catalogs: true
I20250112 20:51:45.658422 1186 tablet_replica.cc:331] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:45.658769 1186 raft_consensus.cc:2238] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.659067 1186 raft_consensus.cc:2267] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.660444 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.660624 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.661159 1164 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:45.661300 1164 ts_tablet_manager.cc:1397] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.217s user 0.003s sys 0.000s
I20250112 20:51:45.662694 1164 raft_consensus.cc:357] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.663659 1164 raft_consensus.cc:383] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.663828 1164 raft_consensus.cc:738] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.664144 1164 consensus_queue.cc:260] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.631968 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f75f9480f6684d1cb710a30844e06e0e (DEFAULT_TABLE table=default.Table_b4a1d95f60f74be2a1f424c514f6efc3 [id=14a48087de9e4e7c8532913e4d67058f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.664960 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75f9480f6684d1cb710a30844e06e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.668744 1164 ts_tablet_manager.cc:1428] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:45.668190 875 tablet_service.cc:1467] Processing CreateTablet for tablet f8a6052e51a7476a955e7d0f8a9977fd (DEFAULT_TABLE table=default.Table_1f284beef9d64af2a22525800952a18b [id=1e9b52f662aa489eb9616cc8ad3527be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.670320 1164 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.676280 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a53e0c8faffe4744926220db2ddb69f9" } modify_external_catalogs: true
I20250112 20:51:45.676537 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a53e0c8faffe4744926220db2ddb69f9" } modify_external_catalogs: true
I20250112 20:51:45.676642 1164 tablet_bootstrap.cc:654] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.680325 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:45.682240 983 leader_election.cc:304] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.682679 2563 raft_consensus.cc:3054] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.684980 2563 raft_consensus.cc:2743] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.686218 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d45ae394dc7849ca9b6f6a8bb647dce0" } modify_external_catalogs: true
I20250112 20:51:45.686411 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d45ae394dc7849ca9b6f6a8bb647dce0" } modify_external_catalogs: true
I20250112 20:51:45.690776 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fec3ae6413364d4bb016e523ce9736aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.691308 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fec3ae6413364d4bb016e523ce9736aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.693670 1178 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.695255 1163 tablet_bootstrap.cc:492] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.695694 737 tablet_service.cc:1467] Processing CreateTablet for tablet f75f9480f6684d1cb710a30844e06e0e (DEFAULT_TABLE table=default.Table_b4a1d95f60f74be2a1f424c514f6efc3 [id=14a48087de9e4e7c8532913e4d67058f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.696924 753 tablet_service.cc:1467] Processing CreateTablet for tablet efa3af02669a47bf8b112c68bb1805d8 (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.698901 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75f9480f6684d1cb710a30844e06e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.699355 754 tablet_service.cc:1467] Processing CreateTablet for tablet aeac9a4034d949e3bbec76b9c32642cf (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.699656 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeac9a4034d949e3bbec76b9c32642cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.704366 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa3af02669a47bf8b112c68bb1805d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.699956 2648 raft_consensus.cc:491] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.700874 881 tablet_service.cc:1467] Processing CreateTablet for tablet aeac9a4034d949e3bbec76b9c32642cf (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.705111 2648 raft_consensus.cc:513] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.705312 884 tablet_service.cc:1467] Processing CreateTablet for tablet efa3af02669a47bf8b112c68bb1805d8 (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.705713 2648 leader_election.cc:290] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:45.701671 1186 ts_tablet_manager.cc:1905] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.706380 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeac9a4034d949e3bbec76b9c32642cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.706588 1178 ts_tablet_manager.cc:1397] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.162s user 0.004s sys 0.000s
I20250112 20:51:45.696035 2447 ts_tablet_manager.cc:1918] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.706800 2447 log.cc:1198] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4561956c97824824a55283a8da21929e
I20250112 20:51:45.707120 2447 ts_tablet_manager.cc:1939] T 4561956c97824824a55283a8da21929e P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.708217 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4561956c97824824a55283a8da21929e (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
I20250112 20:51:45.709926 2447 tablet_replica.cc:331] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.710469 2447 raft_consensus.cc:2238] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.710650 2447 raft_consensus.cc:2267] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.711145 2447 ts_tablet_manager.cc:1905] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.702324 1017 tablet_service.cc:1467] Processing CreateTablet for tablet aeac9a4034d949e3bbec76b9c32642cf (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.706367 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.713713 1163 ts_tablet_manager.cc:1397] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.000s
I20250112 20:51:45.714180 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeac9a4034d949e3bbec76b9c32642cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.714471 1163 raft_consensus.cc:357] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.714655 1163 raft_consensus.cc:383] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.714738 1163 raft_consensus.cc:738] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:45.720902 710 consensus_peers.cc:487] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3343e1d7f1d04a498bd0ee00311b6a61. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.720942 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.722929 2521 raft_consensus.cc:695] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:45.723225 2521 consensus_queue.cc:237] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:45.724225 710 consensus_peers.cc:487] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.707991 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a6052e51a7476a955e7d0f8a9977fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.724642 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa3af02669a47bf8b112c68bb1805d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.705127 885 tablet_service.cc:1467] Processing CreateTablet for tablet f75f9480f6684d1cb710a30844e06e0e (DEFAULT_TABLE table=default.Table_b4a1d95f60f74be2a1f424c514f6efc3 [id=14a48087de9e4e7c8532913e4d67058f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:45.713815 847 leader_election.cc:343] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.725268 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f75f9480f6684d1cb710a30844e06e0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:45.725450 847 leader_election.cc:343] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.725539 847 leader_election.cc:304] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.726012 2802 raft_consensus.cc:2743] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.737260 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c950294baf3046a09d2202912ab8c8c8 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.737689 753 tablet_service.cc:1467] Processing CreateTablet for tablet c950294baf3046a09d2202912ab8c8c8 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.737291 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b496462873e44ceab2ec3c95cf63fc5a (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.738083 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c950294baf3046a09d2202912ab8c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.738438 885 tablet_service.cc:1467] Processing CreateTablet for tablet c950294baf3046a09d2202912ab8c8c8 (DEFAULT_TABLE table=default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3 [id=04247c503770444db479011cf91b0d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.741060 737 tablet_service.cc:1467] Processing CreateTablet for tablet b496462873e44ceab2ec3c95cf63fc5a (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.741401 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b496462873e44ceab2ec3c95cf63fc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.742161 1164 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:45.742274 1164 ts_tablet_manager.cc:1397] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.002s
I20250112 20:51:45.743058 1164 raft_consensus.cc:357] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.743238 1164 raft_consensus.cc:383] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.743496 1178 raft_consensus.cc:357] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.743661 1178 raft_consensus.cc:383] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.743709 1178 raft_consensus.cc:738] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.743907 1178 consensus_queue.cc:260] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.746284 1014 tablet_service.cc:1467] Processing CreateTablet for tablet efa3af02669a47bf8b112c68bb1805d8 (DEFAULT_TABLE table=default.Table_b5c9d97f7893417c9bb1225f856fd737 [id=118829d6220b4f6d9477707fdc841feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.746847 1163 consensus_queue.cc:260] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:45.747565 710 consensus_peers.cc:487] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.753386 1163 ts_tablet_manager.cc:1428] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250112 20:51:45.755713 1185 ts_tablet_manager.cc:1918] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:45.755956 1178 ts_tablet_manager.cc:1428] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20250112 20:51:45.757701 1163 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.758030 1178 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.759683 1163 tablet_bootstrap.cc:654] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.761135 1178 tablet_bootstrap.cc:654] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.743289 1164 raft_consensus.cc:738] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.763808 1164 consensus_queue.cc:260] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.765230 1164 ts_tablet_manager.cc:1428] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
W20250112 20:51:45.777866 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 consensus state has no leader
I20250112 20:51:45.779513 1164 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.780987 1164 tablet_bootstrap.cc:654] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.769276 1185 log.cc:1198] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:45.781848 1185 ts_tablet_manager.cc:1939] T d0018e1eabf24241a358a7e00b6b5907 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:45.782666 1185 tablet_replica.cc:331] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:45.782859 1185 raft_consensus.cc:2238] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:45.783169 1185 raft_consensus.cc:2267] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.783526 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d0018e1eabf24241a358a7e00b6b5907 (table default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]) successfully deleted
I20250112 20:51:45.783879 1185 ts_tablet_manager.cc:1905] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.776178 875 tablet_service.cc:1467] Processing CreateTablet for tablet b496462873e44ceab2ec3c95cf63fc5a (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:45.762480 1049 raft_consensus.cc:3054] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.800900 2521 consensus_queue.cc:1035] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.781829 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.784899 2563 raft_consensus.cc:491] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.803653 2563 raft_consensus.cc:513] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.789753 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c950294baf3046a09d2202912ab8c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.805673 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:45.808954 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ae136d5e2438e950b96f156f23da3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:45.804332 2563 leader_election.cc:290] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.806330 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa3af02669a47bf8b112c68bb1805d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.812265 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b496462873e44ceab2ec3c95cf63fc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.812973 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c950294baf3046a09d2202912ab8c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.813417 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b496462873e44ceab2ec3c95cf63fc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.790048 2447 ts_tablet_manager.cc:1918] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.824920 2447 log.cc:1198] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d706f88f14ab4fd9a459944863afaacd
I20250112 20:51:45.825361 2447 ts_tablet_manager.cc:1939] T d706f88f14ab4fd9a459944863afaacd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.827821 1186 ts_tablet_manager.cc:1918] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.827986 1186 log.cc:1198] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:45.828387 1186 ts_tablet_manager.cc:1939] T 2e6b4882a3d744bda5677aa9e01a53f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:45.830654 983 leader_election.cc:304] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:45.832747 2787 raft_consensus.cc:2743] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:45.836771 2521 consensus_queue.cc:1035] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:45.837679 710 consensus_peers.cc:487] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.837977 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 855) took 2014 ms. Trace:
I20250112 20:51:45.838097 1186 rpcz_store.cc:276] 0112 20:51:43.823812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:43.825641 (+ 1829us) service_pool.cc:225] Handling call
0112 20:51:45.644629 (+1818988us) ts_tablet_manager.cc:1145] Deleting tablet 2e6b4882a3d744bda5677aa9e01a53f4
0112 20:51:45.658335 (+ 13706us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.730506 (+ 72171us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.827717 (+ 97211us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.837963 (+ 10246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818830}
I20250112 20:51:45.838609 2447 tablet_replica.cc:331] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:45.838832 2447 raft_consensus.cc:2238] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:45.838985 2447 raft_consensus.cc:2267] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.840148 2447 ts_tablet_manager.cc:1905] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.843204 2748 raft_consensus.cc:491] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.843335 2748 raft_consensus.cc:513] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.843894 2748 leader_election.cc:290] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:45.844316 710 consensus_peers.cc:487] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:45.845184 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:45.845361 909 raft_consensus.cc:2463] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:45.846009 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:45.846532 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 857) took 1700 ms. Trace:
I20250112 20:51:45.846644 1186 rpcz_store.cc:276] 0112 20:51:44.145796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.145883 (+ 87us) service_pool.cc:225] Handling call
0112 20:51:45.838745 (+1692862us) ts_tablet_manager.cc:1145] Deleting tablet cf6d73ab981c4e87aa229a49fb3b6913
0112 20:51:45.838754 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.846515 (+ 7761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692615}
I20250112 20:51:45.847205 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 858) took 1652 ms. Trace:
I20250112 20:51:45.847323 1186 rpcz_store.cc:276] 0112 20:51:44.194912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.195580 (+ 668us) service_pool.cc:225] Handling call
0112 20:51:45.847101 (+1651521us) ts_tablet_manager.cc:1145] Deleting tablet c23f5f085dd84d68beb277676225e7e2
0112 20:51:45.847110 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.847191 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651328}
I20250112 20:51:45.847736 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 859) took 1621 ms. Trace:
I20250112 20:51:45.847868 1186 rpcz_store.cc:276] 0112 20:51:44.226443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.226599 (+ 156us) service_pool.cc:225] Handling call
0112 20:51:45.847670 (+1621071us) ts_tablet_manager.cc:1145] Deleting tablet 6092a31de9bc47e4bae5f5aebb180c6d
0112 20:51:45.847678 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.847723 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620860}
I20250112 20:51:45.848279 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 860) took 1621 ms. Trace:
I20250112 20:51:45.848405 1186 rpcz_store.cc:276] 0112 20:51:44.226577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.226628 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:45.848215 (+1621587us) ts_tablet_manager.cc:1145] Deleting tablet aa9219fc971c4d15b0d383b20b61e540
0112 20:51:45.848223 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.848267 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621303}
I20250112 20:51:45.848932 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 874) took 1294 ms. Trace:
I20250112 20:51:45.849061 1186 rpcz_store.cc:276] 0112 20:51:44.554416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.557299 (+ 2883us) service_pool.cc:225] Handling call
0112 20:51:45.848764 (+1291465us) ts_tablet_manager.cc:1145] Deleting tablet c45ac747ab7f4990a03a39553424e951
0112 20:51:45.848772 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.848915 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291304}
I20250112 20:51:45.849561 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 875) took 1241 ms. Trace:
I20250112 20:51:45.849689 1186 rpcz_store.cc:276] 0112 20:51:44.608032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.608087 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:45.849456 (+1241369us) ts_tablet_manager.cc:1145] Deleting tablet 9498a04f20dd4cf4a283f17d37ca49ec
0112 20:51:45.849467 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.849547 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241101}
I20250112 20:51:45.850205 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 876) took 1223 ms. Trace:
I20250112 20:51:45.850324 1186 rpcz_store.cc:276] 0112 20:51:44.627005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.627065 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:45.850098 (+1223033us) ts_tablet_manager.cc:1145] Deleting tablet d6a0679bf1ed4bbc85f14f62dfcc114d
0112 20:51:45.850108 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.850190 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222859}
I20250112 20:51:45.850740 1178 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.851109 1178 ts_tablet_manager.cc:1397] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.000s
W20250112 20:51:45.854360 710 leader_election.cc:343] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.854588 710 leader_election.cc:304] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:45.865756 1164 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:45.865967 1164 ts_tablet_manager.cc:1397] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.000s
I20250112 20:51:45.867131 1163 tablet_bootstrap.cc:492] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.867322 1163 ts_tablet_manager.cc:1397] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.001s
I20250112 20:51:45.868103 1163 raft_consensus.cc:357] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.868394 1163 raft_consensus.cc:383] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.868500 1163 raft_consensus.cc:738] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.868774 1163 consensus_queue.cc:260] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.869355 1163 ts_tablet_manager.cc:1428] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:45.850754 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 877) took 1147 ms. Trace:
I20250112 20:51:45.869603 1186 rpcz_store.cc:276] 0112 20:51:44.702904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.702979 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:45.850674 (+1147695us) ts_tablet_manager.cc:1145] Deleting tablet eaa091fe2d3c4f72815f228ac3e39128
0112 20:51:45.850681 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.850743 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147467}
I20250112 20:51:45.870106 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 878) took 1147 ms. Trace:
I20250112 20:51:45.870254 1186 rpcz_store.cc:276] 0112 20:51:44.722826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.722943 (+ 117us) service_pool.cc:225] Handling call
0112 20:51:45.869963 (+1147020us) ts_tablet_manager.cc:1145] Deleting tablet 3b299e16586145ee9ed5a7b118c049b3
0112 20:51:45.869974 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.870088 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146840}
I20250112 20:51:45.870623 1186 tablet_replica.cc:331] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:45.870854 1186 raft_consensus.cc:2238] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:45.871229 1186 raft_consensus.cc:2267] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:45.871985 1186 ts_tablet_manager.cc:1905] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:45.872736 1163 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.874661 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 282eeeb143bc40d899d6c25223582f24
I20250112 20:51:45.875761 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 2e6b4882a3d744bda5677aa9e01a53f4 (table default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]) successfully deleted
I20250112 20:51:45.872301 1178 raft_consensus.cc:357] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:45.878993 1178 raft_consensus.cc:383] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.879127 1178 raft_consensus.cc:738] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.875373 450 catalog_manager.cc:5526] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:45.879289 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cf6d73ab981c4e87aa229a49fb3b6913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf6d73ab981c4e87aa229a49fb3b6913 already in progress: creating tablet
I20250112 20:51:45.879429 1178 consensus_queue.cc:260] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:45.880031 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c23f5f085dd84d68beb277676225e7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23f5f085dd84d68beb277676225e7e2 already in progress: creating tablet
W20250112 20:51:45.880280 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6092a31de9bc47e4bae5f5aebb180c6d because the tablet was not found. No further retry: Not found: Tablet not found: 6092a31de9bc47e4bae5f5aebb180c6d
I20250112 20:51:45.880306 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet b930069b6f1b49439dbc0a7abd3fba72 (table default.Table_f2b30f50ce0843e2861e9bae518ea27b [id=61af9ec286544b919c9f053b647bc6fe]). Expected version 1 got 0
I20250112 20:51:45.881134 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
W20250112 20:51:45.882314 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet aa9219fc971c4d15b0d383b20b61e540 because the tablet was not found. No further retry: Not found: Tablet not found: aa9219fc971c4d15b0d383b20b61e540
W20250112 20:51:45.882543 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c45ac747ab7f4990a03a39553424e951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c45ac747ab7f4990a03a39553424e951 already in progress: creating tablet
W20250112 20:51:45.882731 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9498a04f20dd4cf4a283f17d37ca49ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9498a04f20dd4cf4a283f17d37ca49ec already in progress: creating tablet
W20250112 20:51:45.882908 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d6a0679bf1ed4bbc85f14f62dfcc114d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a0679bf1ed4bbc85f14f62dfcc114d already in progress: creating tablet
I20250112 20:51:45.883085 1178 ts_tablet_manager.cc:1428] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
W20250112 20:51:45.883091 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet eaa091fe2d3c4f72815f228ac3e39128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaa091fe2d3c4f72815f228ac3e39128 already in progress: creating tablet
W20250112 20:51:45.883276 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b299e16586145ee9ed5a7b118c049b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b299e16586145ee9ed5a7b118c049b3 already in progress: creating tablet
I20250112 20:51:45.883482 1178 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.883719 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d706f88f14ab4fd9a459944863afaacd (table default.Table_6289057c9f4f4cb4a78a2ebeb5a048a9 [id=b28cab535c144503b31438bf25d5c2fe]) successfully deleted
I20250112 20:51:45.886583 1163 tablet_bootstrap.cc:654] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.887290 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet be1f12369c894682b7d30de4e37bd466
I20250112 20:51:45.888056 1178 tablet_bootstrap.cc:654] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.889967 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a10f7a47089a43d8ae9c2fc16d379740
I20250112 20:51:45.890367 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2aef567a7cf64178b53625ac4638048b
I20250112 20:51:45.890856 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 53518b453d224ca186fcc6b2ffab5f04 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.892053 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e132d9ac027d44bc8aa20fbdacd57a62" } modify_external_catalogs: true
I20250112 20:51:45.892587 1185 ts_tablet_manager.cc:1918] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:45.892705 1185 log.cc:1198] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d7117edf078848c0a1c83ab8e804dfed
I20250112 20:51:45.893090 1185 ts_tablet_manager.cc:1939] T d7117edf078848c0a1c83ab8e804dfed P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:45.893412 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2ce57ea2a0d44a2d817812e80fe5d573" } new_table_name: "default.Table_a4d34ba24fde4c5fa43588f6f649f3f8" modify_external_catalogs: true
I20250112 20:51:45.893613 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:45.894419 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_09a9e02cfb4c4688afd3b2a20584f610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:45.894800 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2ac8449ec8b94e0494cceb60303f0481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.894964 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09a9e02cfb4c4688afd3b2a20584f610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:45.895264 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ac8449ec8b94e0494cceb60303f0481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.896354 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_65e8cf7dc1134afaae922506cf15579b" } new_table_name: "default.Table_d4e052ce75a047f98d55947e6cb75b67" modify_external_catalogs: true
I20250112 20:51:45.897545 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d95d780f503949e3b24983e2c1202f02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.898011 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d95d780f503949e3b24983e2c1202f02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.892249 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e132d9ac027d44bc8aa20fbdacd57a62" } modify_external_catalogs: true
I20250112 20:51:45.903796 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f75dca2565be402990f84a4a1afe5179" } new_table_name: "default.Table_a3f4e5dfcb9b493cae7e893eace64a9e" modify_external_catalogs: true
I20250112 20:51:45.905652 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b70a9a41263b4b2ea19447b0d87b0efc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.906133 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b70a9a41263b4b2ea19447b0d87b0efc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.907426 2632 raft_consensus.cc:2798] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:45.907513 2632 raft_consensus.cc:491] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:45.907594 2632 raft_consensus.cc:3054] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.907991 1164 raft_consensus.cc:357] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.908151 1164 raft_consensus.cc:383] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.908193 1164 raft_consensus.cc:738] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.908349 1164 consensus_queue.cc:260] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.908854 1164 ts_tablet_manager.cc:1428] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250112 20:51:45.909130 1164 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:45.911026 1164 tablet_bootstrap.cc:654] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:45.912235 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 57660.5us
I20250112 20:51:45.912922 2750 consensus_queue.cc:1035] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:45.915658 1178 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.915797 1178 ts_tablet_manager.cc:1397] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.003s
I20250112 20:51:45.916483 1178 raft_consensus.cc:357] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.916652 1178 raft_consensus.cc:383] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.916703 1178 raft_consensus.cc:738] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.916850 1163 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.916954 1178 consensus_queue.cc:260] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.917537 1178 ts_tablet_manager.cc:1428] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:45.917049 1163 ts_tablet_manager.cc:1397] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250112 20:51:45.919472 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_673a2a9243984736b6b43018eb3e1a42" } new_table_name: "default.Table_3591211d8ed5415b820beae2848ac1f5" modify_external_catalogs: true
I20250112 20:51:45.921656 1178 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:45.923290 1178 tablet_bootstrap.cc:654] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.918859 1163 raft_consensus.cc:357] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.923560 1163 raft_consensus.cc:383] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.923653 1163 raft_consensus.cc:738] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.923904 1163 consensus_queue.cc:260] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.924437 1163 ts_tablet_manager.cc:1428] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 20:51:45.927269 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet be1f12369c894682b7d30de4e37bd466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.927510 737 tablet_service.cc:1514] Processing DeleteTablet for tablet be1f12369c894682b7d30de4e37bd466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.926820 875 tablet_service.cc:1514] Processing DeleteTablet for tablet be1f12369c894682b7d30de4e37bd466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.928870 737 tablet_service.cc:1514] Processing DeleteTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.929087 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.929569 875 tablet_service.cc:1514] Processing DeleteTablet for tablet a10f7a47089a43d8ae9c2fc16d379740 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.930240 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aef567a7cf64178b53625ac4638048b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:45.930581 1163 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:45.932708 2521 raft_consensus.cc:491] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:45.932852 2521 raft_consensus.cc:513] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.933437 2521 leader_election.cc:290] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:45.933969 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:45.936574 847 leader_election.cc:343] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:45.938153 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aef567a7cf64178b53625ac4638048b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:45.938570 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aef567a7cf64178b53625ac4638048b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:45.945552 1163 tablet_bootstrap.cc:654] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.946388 1178 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.946506 1178 ts_tablet_manager.cc:1397] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.003s
I20250112 20:51:45.947146 1178 raft_consensus.cc:357] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.947322 1178 raft_consensus.cc:383] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.947366 1178 raft_consensus.cc:738] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.947542 1178 consensus_queue.cc:260] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:45.948557 1178 ts_tablet_manager.cc:1428] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:45.948876 1178 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:45.950062 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 263b370f96044305a79ebdb79d5e01d1 consensus state has no leader
I20250112 20:51:45.950460 1178 tablet_bootstrap.cc:654] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.950512 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6d10488a4e9d43dea17391673fb9db7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:45.951030 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d10488a4e9d43dea17391673fb9db7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:45.952561 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 734734ff9c564f769bea0984a9e0386b from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:45.952726 1163 tablet_bootstrap.cc:492] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.952868 1163 ts_tablet_manager.cc:1397] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250112 20:51:45.953591 1163 raft_consensus.cc:357] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.953785 1163 raft_consensus.cc:383] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.953826 1163 raft_consensus.cc:738] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.965179 1163 consensus_queue.cc:260] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.965749 1163 ts_tablet_manager.cc:1428] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250112 20:51:45.966037 1163 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:45.978114 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e584b761d234c5aa0ab875acd8e1e05 consensus state has no leader
I20250112 20:51:45.978942 1163 tablet_bootstrap.cc:654] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.979287 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a2dbdbf416f64011b431f5188e0a06c4
I20250112 20:51:45.980351 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 53518b453d224ca186fcc6b2ffab5f04
I20250112 20:51:45.980770 447 catalog_manager.cc:6596] ReplaceTablet: tablet 53518b453d224ca186fcc6b2ffab5f04 deleted and replaced by tablet c07855f83f484e62b400775f71925857
I20250112 20:51:45.981173 1178 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:45.996208 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 734734ff9c564f769bea0984a9e0386b
I20250112 20:51:45.996606 446 catalog_manager.cc:6596] ReplaceTablet: tablet 734734ff9c564f769bea0984a9e0386b deleted and replaced by tablet b9a9a242579444a1aec77dd3b0e58e8f
I20250112 20:51:45.997280 2447 ts_tablet_manager.cc:1918] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:45.997406 2447 log.cc:1198] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:45.997695 2447 ts_tablet_manager.cc:1939] T efa7f6a1e9804e6ab822f9d4c004b1b3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:45.998154 1186 ts_tablet_manager.cc:1918] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:45.998256 1186 log.cc:1198] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/30156185f6104eada87b936fc1829df9
I20250112 20:51:45.998301 1163 tablet_bootstrap.cc:492] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:45.998435 1163 ts_tablet_manager.cc:1397] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.005s
I20250112 20:51:45.998571 1186 ts_tablet_manager.cc:1939] T 30156185f6104eada87b936fc1829df9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:45.999254 1163 raft_consensus.cc:357] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:45.999430 1163 raft_consensus.cc:383] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.999475 1163 raft_consensus.cc:738] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.999665 1163 consensus_queue.cc:260] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.000136 2447 tablet_replica.cc:331] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:46.000183 1163 ts_tablet_manager.cc:1428] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:46.000360 2447 raft_consensus.cc:2238] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.000439 1163 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.000499 2447 raft_consensus.cc:2267] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.001744 2447 ts_tablet_manager.cc:1905] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.002626 1163 tablet_bootstrap.cc:654] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.003872 2632 raft_consensus.cc:513] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.004446 2632 leader_election.cc:290] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.010058 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:46.010535 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:46.010921 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 879) took 1260 ms. Trace:
I20250112 20:51:46.011031 1186 rpcz_store.cc:276] 0112 20:51:44.750051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.750498 (+ 447us) service_pool.cc:225] Handling call
0112 20:51:45.870581 (+1120083us) ts_tablet_manager.cc:1145] Deleting tablet 30156185f6104eada87b936fc1829df9
0112 20:51:45.870590 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.915786 (+ 45196us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.998035 (+ 82249us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.010868 (+ 12833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119923}
I20250112 20:51:46.011466 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 880) took 1251 ms. Trace:
I20250112 20:51:46.011529 1186 rpcz_store.cc:276] 0112 20:51:44.759578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.759625 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:46.011346 (+1251721us) ts_tablet_manager.cc:1145] Deleting tablet 8098069639a2435ab500908db9ecd12a
0112 20:51:46.011358 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.011452 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251570}
I20250112 20:51:46.011828 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 881) took 1213 ms. Trace:
I20250112 20:51:46.011919 1186 rpcz_store.cc:276] 0112 20:51:44.798325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.798384 (+ 59us) service_pool.cc:225] Handling call
0112 20:51:46.011735 (+1213351us) ts_tablet_manager.cc:1145] Deleting tablet 29c4b2cf79d9424eaa2ae8169b4d22a5
0112 20:51:46.011744 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.011815 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213154}
I20250112 20:51:46.012223 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 886) took 1100 ms. Trace:
I20250112 20:51:46.012288 1186 rpcz_store.cc:276] 0112 20:51:44.911631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.911676 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:46.012154 (+1100478us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:46.012162 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.012209 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100298}
I20250112 20:51:46.012503 1186 tablet_replica.cc:331] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:46.012681 1186 raft_consensus.cc:2238] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.012856 1186 raft_consensus.cc:2267] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.014287 1186 ts_tablet_manager.cc:1905] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.015550 446 master_service.cc:946] ReplaceTablet: received request to replace tablet c23f5f085dd84d68beb277676225e7e2 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:46.016469 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d85e56502a8649b19478d46868000abc" } new_table_name: "default.Table_31860fc0efcd4316aa72e27eb6c6c286" modify_external_catalogs: true
I20250112 20:51:46.018523 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:46.022311 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb5331ad9c544514be691d1c1cd10b9a consensus state has no leader
I20250112 20:51:46.026468 875 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.026768 450 master_service.cc:946] ReplaceTablet: received request to replace tablet e92d58058511437d830a8f3cbba1e206 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:46.026897 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dbdbf416f64011b431f5188e0a06c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.028899 2521 raft_consensus.cc:491] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.029042 2521 raft_consensus.cc:513] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.029651 2521 leader_election.cc:290] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.029901 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 with delete_type TABLET_DATA_DELETED (replaced by tablet c07855f83f484e62b400775f71925857) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.030601 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.031173 847 leader_election.cc:343] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.031795 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 885) took 1084 ms. Trace:
I20250112 20:51:46.031880 1185 rpcz_store.cc:276] 0112 20:51:44.947508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.962325 (+ 14817us) service_pool.cc:225] Handling call
0112 20:51:45.782626 (+820301us) ts_tablet_manager.cc:1145] Deleting tablet d7117edf078848c0a1c83ab8e804dfed
0112 20:51:45.782634 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:45.787414 (+ 4780us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:45.892476 (+105062us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.031778 (+139302us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137927,"tablet-delete.queue_time_us":820051}
I20250112 20:51:46.032702 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 with delete_type TABLET_DATA_DELETED (replaced by tablet c07855f83f484e62b400775f71925857) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.034130 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 53518b453d224ca186fcc6b2ffab5f04 with delete_type TABLET_DATA_DELETED (replaced by tablet c07855f83f484e62b400775f71925857) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.040280 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 734734ff9c564f769bea0984a9e0386b with delete_type TABLET_DATA_DELETED (replaced by tablet b9a9a242579444a1aec77dd3b0e58e8f) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.040419 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 890) took 1077 ms. Trace:
I20250112 20:51:46.040513 1185 rpcz_store.cc:276] 0112 20:51:44.962946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.962997 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:46.040319 (+1077322us) ts_tablet_manager.cc:1145] Deleting tablet 2ab635fec8a64efd8e0c527d913ccd7d
0112 20:51:46.040331 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.040401 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077171}
I20250112 20:51:46.042289 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 734734ff9c564f769bea0984a9e0386b with delete_type TABLET_DATA_DELETED (replaced by tablet b9a9a242579444a1aec77dd3b0e58e8f) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.042315 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 734734ff9c564f769bea0984a9e0386b with delete_type TABLET_DATA_DELETED (replaced by tablet b9a9a242579444a1aec77dd3b0e58e8f) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.051931 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:46.052148 909 raft_consensus.cc:3054] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.059363 1178 ts_tablet_manager.cc:1397] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.111s user 0.017s sys 0.013s
I20250112 20:51:46.061043 1185 tablet_replica.cc:331] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:46.061259 1185 raft_consensus.cc:2238] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:46.061595 1185 raft_consensus.cc:2267] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.062099 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e92d58058511437d830a8f3cbba1e206
I20250112 20:51:46.062570 450 catalog_manager.cc:6596] ReplaceTablet: tablet e92d58058511437d830a8f3cbba1e206 deleted and replaced by tablet 01f6a33e6c95485db80f69fab60d5c9b
I20250112 20:51:46.060035 1178 raft_consensus.cc:357] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.063207 1178 raft_consensus.cc:383] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.063264 1178 raft_consensus.cc:738] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.063752 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:46.063894 1178 consensus_queue.cc:260] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.064950 1178 ts_tablet_manager.cc:1428] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20250112 20:51:46.065582 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 30156185f6104eada87b936fc1829df9 (table default.Table_7ca72a25393a40d9bdbbb1d078b7cff2 [id=ecdee533f4374b5b88f11e31a093fb79]) successfully deleted
I20250112 20:51:46.065274 1178 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:46.072084 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8098069639a2435ab500908db9ecd12a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8098069639a2435ab500908db9ecd12a already in progress: creating tablet
W20250112 20:51:46.072351 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c4b2cf79d9424eaa2ae8169b4d22a5 already in progress: creating tablet
W20250112 20:51:46.072587 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2ab635fec8a64efd8e0c527d913ccd7d because the tablet was not found. No further retry: Not found: Tablet not found: 2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:46.072994 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d7117edf078848c0a1c83ab8e804dfed (table default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]) successfully deleted
I20250112 20:51:46.073125 1178 tablet_bootstrap.cc:654] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:46.073257 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2ab635fec8a64efd8e0c527d913ccd7d because the tablet was not found. No further retry: Not found: Tablet not found: 2ab635fec8a64efd8e0c527d913ccd7d
I20250112 20:51:46.073460 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet efa7f6a1e9804e6ab822f9d4c004b1b3 (table default.Table_3407e578085546e59a593ea9e2016ebf [id=605ecbd355e94a3482c85daf985e35fd]) successfully deleted
W20250112 20:51:46.073697 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e8bea4d83ea644a0930a29f61e7a5fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8bea4d83ea644a0930a29f61e7a5fc8 already in progress: creating tablet
I20250112 20:51:46.066625 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.074761 774 raft_consensus.cc:2463] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:46.067306 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.075532 1185 ts_tablet_manager.cc:1905] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:46.077490 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet cc90d3722fd14d59b9994655cd30ef2b consensus state has no leader
I20250112 20:51:46.078691 1164 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.078877 1164 ts_tablet_manager.cc:1397] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.170s user 0.021s sys 0.017s
I20250112 20:51:46.080173 1164 raft_consensus.cc:357] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.080415 1164 raft_consensus.cc:383] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.080509 1164 raft_consensus.cc:738] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.080750 1164 consensus_queue.cc:260] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.081498 1164 ts_tablet_manager.cc:1428] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.081791 1164 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.085808 1178 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.085951 1178 ts_tablet_manager.cc:1397] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250112 20:51:46.086726 1164 tablet_bootstrap.cc:654] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.086643 1178 raft_consensus.cc:357] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.086810 1178 raft_consensus.cc:383] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.086871 1178 raft_consensus.cc:738] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.087073 1178 consensus_queue.cc:260] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.092718 1178 ts_tablet_manager.cc:1428] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:46.093030 1178 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.094441 1178 tablet_bootstrap.cc:654] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.097466 2521 raft_consensus.cc:491] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.097589 2521 raft_consensus.cc:513] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.098166 2521 leader_election.cc:290] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.098842 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.099413 847 leader_election.cc:343] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:46.100383 710 leader_election.cc:343] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.100929 909 raft_consensus.cc:2463] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:46.101068 2632 consensus_queue.cc:1035] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.096735 2447 ts_tablet_manager.cc:1918] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.101563 2447 log.cc:1198] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d7117edf078848c0a1c83ab8e804dfed
I20250112 20:51:46.102057 2447 ts_tablet_manager.cc:1939] T d7117edf078848c0a1c83ab8e804dfed P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:46.105428 710 leader_election.cc:304] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.106292 2632 raft_consensus.cc:2798] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:46.107062 1163 tablet_bootstrap.cc:492] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.107257 1163 ts_tablet_manager.cc:1397] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.021s
I20250112 20:51:46.107296 2632 raft_consensus.cc:695] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:46.107545 2632 consensus_queue.cc:237] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.108208 1163 raft_consensus.cc:357] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.108434 1163 raft_consensus.cc:383] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.108559 1163 raft_consensus.cc:738] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.108934 1163 consensus_queue.cc:260] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.109686 1163 ts_tablet_manager.cc:1428] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.110044 1163 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.112685 1163 tablet_bootstrap.cc:654] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.118475 1178 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.118613 1178 ts_tablet_manager.cc:1397] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250112 20:51:46.119187 1163 tablet_bootstrap.cc:492] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.119313 1163 ts_tablet_manager.cc:1397] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250112 20:51:46.119302 1178 raft_consensus.cc:357] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.119490 1178 raft_consensus.cc:383] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.119575 1178 raft_consensus.cc:738] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.119807 1178 consensus_queue.cc:260] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.120069 1163 raft_consensus.cc:357] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.120262 1163 raft_consensus.cc:383] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.120342 1163 raft_consensus.cc:738] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.120514 1163 consensus_queue.cc:260] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.121017 1163 ts_tablet_manager.cc:1428] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.123023 1178 ts_tablet_manager.cc:1428] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:46.123510 1178 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.125046 1163 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.126080 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d58058511437d830a8f3cbba1e206 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f6a33e6c95485db80f69fab60d5c9b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.127889 875 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d58058511437d830a8f3cbba1e206 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f6a33e6c95485db80f69fab60d5c9b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.131091 1164 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.131232 1164 ts_tablet_manager.cc:1397] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.004s
I20250112 20:51:46.131528 741 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d58058511437d830a8f3cbba1e206 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f6a33e6c95485db80f69fab60d5c9b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.131958 1164 raft_consensus.cc:357] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.132124 1164 raft_consensus.cc:383] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.132184 1164 raft_consensus.cc:738] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.132345 1164 consensus_queue.cc:260] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.132840 1164 ts_tablet_manager.cc:1428] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.136740 1178 tablet_bootstrap.cc:654] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.145228 1163 tablet_bootstrap.cc:654] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.147396 847 leader_election.cc:304] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.147763 2521 raft_consensus.cc:2798] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.147866 2521 raft_consensus.cc:491] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.147935 2521 raft_consensus.cc:3054] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:46.152002 847 leader_election.cc:343] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.152124 847 leader_election.cc:304] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.152474 2802 raft_consensus.cc:2743] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.153472 1178 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.153594 1178 ts_tablet_manager.cc:1397] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20250112 20:51:46.154454 1178 raft_consensus.cc:357] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.154691 1178 raft_consensus.cc:383] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.154753 1178 raft_consensus.cc:738] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.155023 1178 consensus_queue.cc:260] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.162406 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0515c2f2d9b346fe99c0cea8663c500a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.163458 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0515c2f2d9b346fe99c0cea8663c500a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.167497 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_132455edcc094422bdf769a1c836cf16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.168020 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_132455edcc094422bdf769a1c836cf16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.170236 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2784d0f26fb740fe894e7750b460886d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.170693 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2784d0f26fb740fe894e7750b460886d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.172426 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.173640 1164 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.174796 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1175f36a82364f40bf29fd45784c07a0" } modify_external_catalogs: true
I20250112 20:51:46.175150 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1175f36a82364f40bf29fd45784c07a0" } modify_external_catalogs: true
I20250112 20:51:46.175536 1164 tablet_bootstrap.cc:654] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.176754 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
W20250112 20:51:46.178455 847 leader_election.cc:343] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.178611 847 leader_election.cc:304] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.179134 2802 raft_consensus.cc:2743] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.179971 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1f59b364b3b3490aaa782d7e83f048ff" } modify_external_catalogs: true
I20250112 20:51:46.180521 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_527edcc30df04186a34eb7b085689535" } modify_external_catalogs: true
I20250112 20:51:46.181496 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1f59b364b3b3490aaa782d7e83f048ff" } modify_external_catalogs: true
I20250112 20:51:46.181519 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_527edcc30df04186a34eb7b085689535" } modify_external_catalogs: true
I20250112 20:51:46.183424 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 859) took 1046 ms. Trace:
I20250112 20:51:46.183518 2447 rpcz_store.cc:276] 0112 20:51:45.137147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.137207 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:46.000095 (+862888us) ts_tablet_manager.cc:1145] Deleting tablet d7117edf078848c0a1c83ab8e804dfed
0112 20:51:46.000103 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.092957 (+ 92854us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.096628 (+ 3671us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.183374 (+ 86746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":862678}
I20250112 20:51:46.183887 2447 tablet_replica.cc:331] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:46.184154 2447 raft_consensus.cc:2238] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.184298 2447 raft_consensus.cc:2267] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.184871 2447 ts_tablet_manager.cc:1905] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.185108 1178 ts_tablet_manager.cc:1428] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20250112 20:51:46.185452 1178 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.186980 1178 tablet_bootstrap.cc:654] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.189370 1186 ts_tablet_manager.cc:1918] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.189468 1186 log.cc:1198] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d7117edf078848c0a1c83ab8e804dfed
I20250112 20:51:46.189783 1186 ts_tablet_manager.cc:1939] T d7117edf078848c0a1c83ab8e804dfed P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:46.191879 2521 raft_consensus.cc:513] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.192510 2521 leader_election.cc:290] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.193681 1163 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.193877 1163 ts_tablet_manager.cc:1397] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.005s
I20250112 20:51:46.194677 1163 raft_consensus.cc:357] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.194875 1163 raft_consensus.cc:383] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.194928 1163 raft_consensus.cc:738] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.195101 1163 consensus_queue.cc:260] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.195571 1163 ts_tablet_manager.cc:1428] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.195838 1163 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.197535 1163 tablet_bootstrap.cc:654] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.193681 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:46.202831 847 leader_election.cc:343] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.205214 1164 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.205365 1164 ts_tablet_manager.cc:1397] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.006s
I20250112 20:51:46.206132 1164 raft_consensus.cc:357] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.206316 1164 raft_consensus.cc:383] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.206439 1164 raft_consensus.cc:738] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.206670 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c6bdfecd5d9142d5b008c613864c8217
I20250112 20:51:46.206629 1164 consensus_queue.cc:260] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.207132 1164 ts_tablet_manager.cc:1428] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.207386 1164 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.207639 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f0fb6a5146c44edea076a10da58ea1a7
I20250112 20:51:46.209378 1164 tablet_bootstrap.cc:654] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.223920 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f143227e79844a7393b323222bf7eaef (table default.Table_d4e052ce75a047f98d55947e6cb75b67 [id=467b97b4e4694b2f8551e237429bd0d3]). Expected version 1 got 0
I20250112 20:51:46.224977 1164 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.225121 1164 ts_tablet_manager.cc:1397] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250112 20:51:46.229087 1164 raft_consensus.cc:357] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.224725 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f81091f6a28b413a91f2af4ff8f05df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:46.229278 1164 raft_consensus.cc:383] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.229327 1164 raft_consensus.cc:738] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.229493 1164 consensus_queue.cc:260] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:46.229669 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f81091f6a28b413a91f2af4ff8f05df5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.225688 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11e0da2a1beb4196933adeeb88dfed4a" } modify_external_catalogs: true
I20250112 20:51:46.229168 453 master_service.cc:946] ReplaceTablet: received request to replace tablet f424288fe62849edbbd2d38e89331fb7 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:46.245074 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11e0da2a1beb4196933adeeb88dfed4a" } modify_external_catalogs: true
I20250112 20:51:46.247592 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f424288fe62849edbbd2d38e89331fb7
I20250112 20:51:46.248034 453 catalog_manager.cc:6596] ReplaceTablet: tablet f424288fe62849edbbd2d38e89331fb7 deleted and replaced by tablet 754641e1c24b45faaabfb6c001d4647a
I20250112 20:51:46.254426 2845 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:46.257998 446 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:46.259734 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 889) took 1347 ms. Trace:
I20250112 20:51:46.259840 1186 rpcz_store.cc:276] 0112 20:51:44.911924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.911961 (+ 37us) service_pool.cc:225] Handling call
0112 20:51:46.012464 (+1100503us) ts_tablet_manager.cc:1145] Deleting tablet d7117edf078848c0a1c83ab8e804dfed
0112 20:51:46.012473 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.096960 (+ 84487us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.189261 (+ 92301us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.259716 (+ 70455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100358}
I20250112 20:51:46.260123 1186 tablet_replica.cc:331] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:46.260295 1186 raft_consensus.cc:2238] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.260447 1186 raft_consensus.cc:2267] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.263201 1186 ts_tablet_manager.cc:1905] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.263856 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:46.264034 775 raft_consensus.cc:3054] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.266594 775 raft_consensus.cc:2463] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:46.267145 2447 ts_tablet_manager.cc:1918] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:46.267266 2447 log.cc:1198] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9856e6b625d04f17b0869cee08f69142
I20250112 20:51:46.267422 2563 raft_consensus.cc:491] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.267578 2447 ts_tablet_manager.cc:1939] T 9856e6b625d04f17b0869cee08f69142 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:46.267549 2563 raft_consensus.cc:513] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.267647 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a3f4e5dfcb9b493cae7e893eace64a9e" } new_table_name: "default.Table_f257ef99d9ec49d99f2b4df753f21d4e" modify_external_catalogs: true
I20250112 20:51:46.268198 2563 leader_election.cc:290] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.268337 2846 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:46.268492 1164 ts_tablet_manager.cc:1428] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250112 20:51:46.268791 1164 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.268771 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.268986 909 raft_consensus.cc:2463] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:46.269325 983 leader_election.cc:304] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:46.269623 2563 raft_consensus.cc:2798] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.269698 2563 raft_consensus.cc:491] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.269765 2563 raft_consensus.cc:3054] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.270365 1164 tablet_bootstrap.cc:654] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.274859 2521 raft_consensus.cc:491] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.275003 2521 raft_consensus.cc:513] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.275575 2521 leader_election.cc:290] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.278299 1164 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.278565 1164 ts_tablet_manager.cc:1397] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250112 20:51:46.278980 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet f143227e79844a7393b323222bf7eaef (table default.Table_d4e052ce75a047f98d55947e6cb75b67 [id=467b97b4e4694b2f8551e237429bd0d3]). Expected version 1 got 0
I20250112 20:51:46.279213 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.279338 1164 raft_consensus.cc:357] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.279498 1164 raft_consensus.cc:383] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.279556 1164 raft_consensus.cc:738] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:46.279881 847 leader_election.cc:343] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.279853 1164 consensus_queue.cc:260] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.280700 1164 ts_tablet_manager.cc:1428] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:46.281431 1164 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.283409 1164 tablet_bootstrap.cc:654] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:46.288548 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e584b761d234c5aa0ab875acd8e1e05 consensus state has no leader
I20250112 20:51:46.288743 2521 raft_consensus.cc:491] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.288944 2521 raft_consensus.cc:513] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.289695 2521 leader_election.cc:290] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.290119 1178 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.290256 1178 ts_tablet_manager.cc:1397] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.105s user 0.015s sys 0.011s
I20250112 20:51:46.290817 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.290894 1178 raft_consensus.cc:357] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.291093 1178 raft_consensus.cc:383] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.291170 1178 raft_consensus.cc:738] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.291405 1178 consensus_queue.cc:260] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.291988 1178 ts_tablet_manager.cc:1428] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 20:51:46.292301 847 leader_election.cc:343] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.305169 2059 raft_consensus.cc:491] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.305320 2059 raft_consensus.cc:513] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.305867 2059 leader_election.cc:290] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.306856 1178 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.308483 1178 tablet_bootstrap.cc:654] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.312109 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_53972fb4ced04252bb122116083a5c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.312920 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53972fb4ced04252bb122116083a5c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.317823 1163 tablet_bootstrap.cc:492] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.318032 1163 ts_tablet_manager.cc:1397] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.122s user 0.009s sys 0.010s
I20250112 20:51:46.318874 1163 raft_consensus.cc:357] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.319046 1163 raft_consensus.cc:383] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.319087 1163 raft_consensus.cc:738] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.320398 2787 raft_consensus.cc:491] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.320554 2787 raft_consensus.cc:513] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.321282 2787 leader_election.cc:290] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.321374 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_28ec28b7ea3048b6adc23abf2325689a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.321869 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28ec28b7ea3048b6adc23abf2325689a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.322153 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.322162 1163 consensus_queue.cc:260] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.323585 1163 ts_tablet_manager.cc:1428] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:46.324990 1163 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.326448 1163 tablet_bootstrap.cc:654] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.332316 1164 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.332446 1164 ts_tablet_manager.cc:1397] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.007s
I20250112 20:51:46.333220 1164 raft_consensus.cc:357] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.333406 1164 raft_consensus.cc:383] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.333448 1164 raft_consensus.cc:738] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.333611 1164 consensus_queue.cc:260] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.334177 1164 ts_tablet_manager.cc:1428] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.334481 1164 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.336247 1164 tablet_bootstrap.cc:654] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.349562 1163 tablet_bootstrap.cc:492] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.349718 1163 ts_tablet_manager.cc:1397] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250112 20:51:46.350656 1163 raft_consensus.cc:357] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.350912 1163 raft_consensus.cc:383] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.350961 1163 raft_consensus.cc:738] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.351233 1163 consensus_queue.cc:260] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.352792 1163 ts_tablet_manager.cc:1428] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:46.353825 1163 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.354693 2563 raft_consensus.cc:513] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.355238 2563 leader_election.cc:290] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.357193 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b96c62442b9e4c6a8ee4b88a6a990e6f" } new_table_name: "default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf" modify_external_catalogs: true
I20250112 20:51:46.357450 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:46.357616 909 raft_consensus.cc:3054] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.360407 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 860) took 1222 ms. Trace:
I20250112 20:51:46.360495 2447 rpcz_store.cc:276] 0112 20:51:45.137607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.140484 (+ 2877us) service_pool.cc:225] Handling call
0112 20:51:46.183825 (+1043341us) ts_tablet_manager.cc:1145] Deleting tablet 9856e6b625d04f17b0869cee08f69142
0112 20:51:46.183836 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.187742 (+ 3906us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.267028 (+ 79286us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.360386 (+ 93358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043139}
I20250112 20:51:46.361708 2447 tablet_replica.cc:331] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:46.361909 2447 raft_consensus.cc:2238] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.362035 2447 raft_consensus.cc:2267] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.364552 2447 ts_tablet_manager.cc:1905] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:46.365383 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d72ae28879dc45429de2ab0d66276255 consensus state has no leader
I20250112 20:51:46.369033 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b9a0b9c1271a440f95be03d7aff157ef" } new_table_name: "default.Table_3ac63da2597b4f6a87e1619e911a5ee7" modify_external_catalogs: true
I20250112 20:51:46.370308 1164 tablet_bootstrap.cc:492] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.370450 1164 ts_tablet_manager.cc:1397] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.005s
I20250112 20:51:46.371083 1164 raft_consensus.cc:357] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.371266 1164 raft_consensus.cc:383] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.371322 1164 raft_consensus.cc:738] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.371552 1164 consensus_queue.cc:260] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.372411 1164 ts_tablet_manager.cc:1428] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:46.372588 1163 tablet_bootstrap.cc:654] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.372805 1164 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.373844 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0ede004124b94194a25701ee39ebfc60" } modify_external_catalogs: true
I20250112 20:51:46.374766 1164 tablet_bootstrap.cc:654] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.375439 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0ede004124b94194a25701ee39ebfc60" } modify_external_catalogs: true
I20250112 20:51:46.376569 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_638524600a3c447b90e0c84bf8d929c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.377264 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_638524600a3c447b90e0c84bf8d929c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:46.378494 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b1edc79f17f4bb7a6bb9ab619403142 consensus state has no leader
I20250112 20:51:46.386572 2059 consensus_queue.cc:1035] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.387445 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_73ac32d005e74ff68df62ba8e7cc94ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.387954 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73ac32d005e74ff68df62ba8e7cc94ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.389307 1164 tablet_bootstrap.cc:492] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.389418 1164 ts_tablet_manager.cc:1397] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250112 20:51:46.392517 2521 raft_consensus.cc:491] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.392693 2521 raft_consensus.cc:513] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.393462 1163 tablet_bootstrap.cc:492] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.393596 1163 ts_tablet_manager.cc:1397] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250112 20:51:46.393941 2521 leader_election.cc:290] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.394054 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.394379 1163 raft_consensus.cc:357] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.394541 1163 raft_consensus.cc:383] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.394579 1163 raft_consensus.cc:738] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:46.394623 847 leader_election.cc:343] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.394744 1163 consensus_queue.cc:260] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.397522 1164 raft_consensus.cc:357] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.397694 1164 raft_consensus.cc:383] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.397742 1164 raft_consensus.cc:738] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.397800 1163 ts_tablet_manager.cc:1428] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:46.397783 2845 tablet.cc:1721] T f143227e79844a7393b323222bf7eaef P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:46.397920 1164 consensus_queue.cc:260] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.398483 1163 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.398387 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_23ec53724daa45fa96c695571826a6ea" } new_table_name: "default.Table_0f89b82fe1324854be88bd57977f1556" modify_external_catalogs: true
I20250112 20:51:46.400189 1163 tablet_bootstrap.cc:654] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:46.404568 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f56b47cebed483fa9679158c9cae314 consensus state has no leader
I20250112 20:51:46.406596 1164 ts_tablet_manager.cc:1428] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250112 20:51:46.406909 1164 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.408646 1164 tablet_bootstrap.cc:654] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.409631 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_106af948e5aa4b87830e03feedeae902"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:46.412683 847 leader_election.cc:304] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.413244 2802 raft_consensus.cc:2798] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:46.413386 2802 raft_consensus.cc:695] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:46.413643 2802 consensus_queue.cc:237] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.413793 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.410275 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2e76748510074c8f882cd75a58be6dbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:46.419059 1049 raft_consensus.cc:3054] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.411518 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eb719443baaa46e49179af022eae1a4b" } new_table_name: "default.Table_cb6c84ab3dd344d28800d498d1816591" modify_external_catalogs: true
I20250112 20:51:46.413380 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 0a87da1ea4054ffc81b9c1c792f0cc5d from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:46.421252 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e76748510074c8f882cd75a58be6dbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:46.422441 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_106af948e5aa4b87830e03feedeae902 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.424407 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.425287 847 consensus_peers.cc:487] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:46.425473 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bdfecd5d9142d5b008c613864c8217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.425526 2563 raft_consensus.cc:491] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.425704 2563 raft_consensus.cc:513] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.426288 2563 leader_election.cc:290] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.426906 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.427084 908 raft_consensus.cc:2463] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:46.427312 2563 raft_consensus.cc:491] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.427453 2563 raft_consensus.cc:513] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.427503 983 leader_election.cc:304] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:46.427855 2787 raft_consensus.cc:2798] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.427963 2787 raft_consensus.cc:491] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.428043 2787 raft_consensus.cc:3054] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.428089 2563 leader_election.cc:290] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.428565 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.445290 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bdfecd5d9142d5b008c613864c8217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:46.446043 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d94c67881e954c0ebecf5c31de665ac0 consensus state has no leader
I20250112 20:51:46.449644 447 catalog_manager.cc:5526] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:46.450022 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.450222 908 raft_consensus.cc:2463] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:46.450299 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:46.453898 875 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bdfecd5d9142d5b008c613864c8217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.454895 1164 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.455087 1164 ts_tablet_manager.cc:1397] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.008s
I20250112 20:51:46.455483 875 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.456149 1164 raft_consensus.cc:357] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.456395 1164 raft_consensus.cc:383] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.456863 1164 raft_consensus.cc:738] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.457226 1164 consensus_queue.cc:260] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.457356 909 raft_consensus.cc:2463] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:46.457815 1186 ts_tablet_manager.cc:1918] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.457901 1164 ts_tablet_manager.cc:1428] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:46.457934 1186 log.cc:1198] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:46.458159 1164 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.458184 983 leader_election.cc:304] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:46.458258 1186 ts_tablet_manager.cc:1939] T d0018e1eabf24241a358a7e00b6b5907 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:46.457818 2802 raft_consensus.cc:491] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.458474 2563 raft_consensus.cc:2798] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:46.458633 2802 raft_consensus.cc:513] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.459740 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.460263 847 leader_election.cc:343] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.460384 1164 tablet_bootstrap.cc:654] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.460474 2802 leader_election.cc:290] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.462054 1185 ts_tablet_manager.cc:1918] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.462193 1185 log.cc:1198] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:46.462630 1185 ts_tablet_manager.cc:1939] T 3343e1d7f1d04a498bd0ee00311b6a61 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:46.465476 1178 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.465582 1178 ts_tablet_manager.cc:1397] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.159s user 0.000s sys 0.003s
I20250112 20:51:46.466243 1178 raft_consensus.cc:357] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.466401 1178 raft_consensus.cc:383] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.466445 1178 raft_consensus.cc:738] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.469004 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 887) took 1557 ms. Trace:
I20250112 20:51:46.469096 1186 rpcz_store.cc:276] 0112 20:51:44.911717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.912340 (+ 623us) service_pool.cc:225] Handling call
0112 20:51:46.260078 (+1347738us) ts_tablet_manager.cc:1145] Deleting tablet d0018e1eabf24241a358a7e00b6b5907
0112 20:51:46.260086 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.354675 (+ 94589us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.457715 (+103040us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.468990 (+ 11275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347614}
I20250112 20:51:46.469373 1186 tablet_replica.cc:331] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:46.469535 1186 raft_consensus.cc:2238] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.469743 1186 raft_consensus.cc:2267] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.470363 1186 ts_tablet_manager.cc:1905] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.471415 1164 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.471524 1164 ts_tablet_manager.cc:1397] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20250112 20:51:46.472172 1164 raft_consensus.cc:357] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.472347 1164 raft_consensus.cc:383] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.472393 1164 raft_consensus.cc:738] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.472563 1164 consensus_queue.cc:260] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:46.472710 847 consensus_peers.cc:487] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:46.472888 2802 raft_consensus.cc:491] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.472987 2802 raft_consensus.cc:513] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.473332 1164 ts_tablet_manager.cc:1428] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:46.473577 1164 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.473732 2802 leader_election.cc:290] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.469523 1178 consensus_queue.cc:260] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.474532 1178 ts_tablet_manager.cc:1428] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250112 20:51:46.474792 1178 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.475234 1164 tablet_bootstrap.cc:654] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.476621 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.477778 847 leader_election.cc:343] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.480355 1178 tablet_bootstrap.cc:654] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.481815 1041 raft_consensus.cc:3054] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.482532 2563 raft_consensus.cc:695] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:46.483362 2563 consensus_queue.cc:237] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.490020 1164 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.490168 1164 ts_tablet_manager.cc:1397] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250112 20:51:46.490826 1164 raft_consensus.cc:357] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.491021 1164 raft_consensus.cc:383] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.491087 1164 raft_consensus.cc:738] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.491351 1164 consensus_queue.cc:260] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.491915 1042 raft_consensus.cc:3054] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.491978 1164 ts_tablet_manager.cc:1428] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:46.492293 1164 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.494273 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.494371 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb6a5146c44edea076a10da58ea1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.496856 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f424288fe62849edbbd2d38e89331fb7 with delete_type TABLET_DATA_DELETED (replaced by tablet 754641e1c24b45faaabfb6c001d4647a) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.497572 1164 tablet_bootstrap.cc:654] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.504060 1178 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.504199 1178 ts_tablet_manager.cc:1397] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20250112 20:51:46.506444 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6158b97ccf0a42fdb1a510c5d1e80213
I20250112 20:51:46.506572 1178 raft_consensus.cc:357] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.506743 1178 raft_consensus.cc:383] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.506783 1178 raft_consensus.cc:738] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.506965 1178 consensus_queue.cc:260] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.507391 1178 ts_tablet_manager.cc:1428] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:46.507477 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8229e4ef771b4032a4bfae3190a851e3
I20250112 20:51:46.508577 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2de03e3114d145aeb81590dbae2c548e
I20250112 20:51:46.509312 2750 raft_consensus.cc:491] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.509446 2750 raft_consensus.cc:513] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.510072 2750 leader_election.cc:290] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.514045 1164 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.514191 1164 ts_tablet_manager.cc:1397] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250112 20:51:46.512389 2865 tablet.cc:1721] T f143227e79844a7393b323222bf7eaef P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:46.515370 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f424288fe62849edbbd2d38e89331fb7 with delete_type TABLET_DATA_DELETED (replaced by tablet 754641e1c24b45faaabfb6c001d4647a) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.515027 875 tablet_service.cc:1514] Processing DeleteTablet for tablet f424288fe62849edbbd2d38e89331fb7 with delete_type TABLET_DATA_DELETED (replaced by tablet 754641e1c24b45faaabfb6c001d4647a) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.515818 1178 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.516315 884 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.516700 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.516880 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.517283 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.517283 892 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.517668 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.518450 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.522152 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d7117edf078848c0a1c83ab8e804dfed (table default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]) successfully deleted
I20250112 20:51:46.522431 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d0018e1eabf24241a358a7e00b6b5907 (table default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]) successfully deleted
I20250112 20:51:46.522656 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d7117edf078848c0a1c83ab8e804dfed (table default.Table_1d3f834543f44a0eb5d7f2ec4a5f351a [id=59653ec2ff7049279462f22765517126]) successfully deleted
I20250112 20:51:46.522878 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9856e6b625d04f17b0869cee08f69142 (table default.Table_17b597f08a264f22b57f12049ca81079 [id=c2e70418c71e48abb3e8b890cdc0ce3a]) successfully deleted
I20250112 20:51:46.523866 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 02c897c943cb4fb2aa4d887054f56d99 (DEFAULT_TABLE table=default.Table_201e5a51990a4629a810d8e3d9226ce7 [id=ddfd1357cb3244f78a12964ac5af2d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.525820 2059 consensus_queue.cc:1035] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:46.526495 710 leader_election.cc:343] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.527750 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.527863 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f9b46da06a476ab9f2a93ca74ff736" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.527989 775 raft_consensus.cc:2463] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:46.528327 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577dc789f0cb48f49168124b37a6fba0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:46.528393 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.528492 775 raft_consensus.cc:3054] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.528512 773 raft_consensus.cc:2463] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:46.528887 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ea7f38f1445c598676715a3036227" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.529381 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.529735 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.529896 773 raft_consensus.cc:2463] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:46.532135 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.534675 2861 raft_consensus.cc:491] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.534853 2861 raft_consensus.cc:513] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.536036 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.536651 2861 leader_election.cc:290] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.541211 1164 raft_consensus.cc:357] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.542028 2863 tablet.cc:1721] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:46.542874 1164 raft_consensus.cc:383] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.543097 1164 raft_consensus.cc:738] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.543581 1164 consensus_queue.cc:260] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:46.545300 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271): Couldn't send request to peer b985fd656ce44131bc91ccfeb79587c5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 20:51:46.548895 1164 ts_tablet_manager.cc:1428] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20250112 20:51:46.550316 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:46.550612 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:46.550868 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:46.553637 1164 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.558009 1164 tablet_bootstrap.cc:654] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.561244 1186 ts_tablet_manager.cc:1918] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:46.561344 1186 log.cc:1198] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/577dc789f0cb48f49168124b37a6fba0
I20250112 20:51:46.561585 1186 ts_tablet_manager.cc:1939] T 577dc789f0cb48f49168124b37a6fba0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:46.563313 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 892) took 1600 ms. Trace:
I20250112 20:51:46.563406 1185 rpcz_store.cc:276] 0112 20:51:44.963154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963193 (+ 39us) service_pool.cc:225] Handling call
0112 20:51:46.060986 (+1097793us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:46.060998 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.221345 (+160347us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.461928 (+240583us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.563295 (+101367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097671}
I20250112 20:51:46.563761 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 893) took 1600 ms. Trace:
I20250112 20:51:46.563836 1185 rpcz_store.cc:276] 0112 20:51:44.963265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963340 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:46.563668 (+1600328us) ts_tablet_manager.cc:1145] Deleting tablet 54ab0ae10f09469fa4f913e049002481
0112 20:51:46.563678 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.563746 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600216}
I20250112 20:51:46.564154 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 895) took 1600 ms. Trace:
I20250112 20:51:46.564221 1185 rpcz_store.cc:276] 0112 20:51:44.963433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963473 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:46.564081 (+1600608us) ts_tablet_manager.cc:1145] Deleting tablet efa7f6a1e9804e6ab822f9d4c004b1b3
0112 20:51:46.564089 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.564141 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600499}
I20250112 20:51:46.564455 1185 tablet_replica.cc:331] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:46.564652 1185 raft_consensus.cc:2238] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:46.566458 2787 raft_consensus.cc:513] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.567152 2787 leader_election.cc:290] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.567445 775 raft_consensus.cc:2463] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:46.569756 1178 tablet_bootstrap.cc:654] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.571189 2447 ts_tablet_manager.cc:1918] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:46.571297 2447 log.cc:1198] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/89f9b46da06a476ab9f2a93ca74ff736
I20250112 20:51:46.571235 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:46.571460 908 raft_consensus.cc:3054] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.571561 2447 ts_tablet_manager.cc:1939] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:46.571779 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c897c943cb4fb2aa4d887054f56d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.576009 1185 raft_consensus.cc:2267] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.576558 1185 ts_tablet_manager.cc:1905] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.581846 1164 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.582006 1164 ts_tablet_manager.cc:1397] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.001s
I20250112 20:51:46.582494 775 raft_consensus.cc:1270] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Refusing update from remote peer 87ae45957c64469d96dc92d606b5b4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:46.582890 1164 raft_consensus.cc:357] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.583076 1164 raft_consensus.cc:383] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.583139 1164 raft_consensus.cc:738] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.583355 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.583350 1164 consensus_queue.cc:260] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.583920 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:46.584084 775 raft_consensus.cc:2463] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:46.585180 1164 ts_tablet_manager.cc:1428] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:46.585491 1164 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.586345 1163 tablet_bootstrap.cc:492] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.586486 1163 ts_tablet_manager.cc:1397] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.188s user 0.026s sys 0.022s
I20250112 20:51:46.587133 1164 tablet_bootstrap.cc:654] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.587052 741 tablet_service.cc:1467] Processing CreateTablet for tablet 02c897c943cb4fb2aa4d887054f56d99 (DEFAULT_TABLE table=default.Table_201e5a51990a4629a810d8e3d9226ce7 [id=ddfd1357cb3244f78a12964ac5af2d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.587194 1163 raft_consensus.cc:357] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.587388 1163 raft_consensus.cc:383] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.587451 1163 raft_consensus.cc:738] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.587628 1163 consensus_queue.cc:260] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.587901 881 tablet_service.cc:1467] Processing CreateTablet for tablet 02c897c943cb4fb2aa4d887054f56d99 (DEFAULT_TABLE table=default.Table_201e5a51990a4629a810d8e3d9226ce7 [id=ddfd1357cb3244f78a12964ac5af2d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.588124 1163 ts_tablet_manager.cc:1428] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.588382 737 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90d6eafea84751b1f1556ca490a4ea (DEFAULT_TABLE table=default.Table_64d69b6c7750480191ca2e8f3234b1e5 [id=bb4b6a35e199408789eaf218773dd769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.588385 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90d6eafea84751b1f1556ca490a4ea (DEFAULT_TABLE table=default.Table_64d69b6c7750480191ca2e8f3234b1e5 [id=bb4b6a35e199408789eaf218773dd769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.588568 1163 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.588770 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90d6eafea84751b1f1556ca490a4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.589936 1163 tablet_bootstrap.cc:654] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.591002 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.591183 909 raft_consensus.cc:2463] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:46.592964 1178 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.593138 1178 ts_tablet_manager.cc:1397] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.011s
I20250112 20:51:46.593094 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.593928 1178 raft_consensus.cc:357] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.594125 1178 raft_consensus.cc:383] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.594189 1178 raft_consensus.cc:738] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.594377 1178 consensus_queue.cc:260] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.594874 1178 ts_tablet_manager.cc:1428] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.597899 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 888) took 1686 ms. Trace:
I20250112 20:51:46.597995 1186 rpcz_store.cc:276] 0112 20:51:44.911800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.912494 (+ 694us) service_pool.cc:225] Handling call
0112 20:51:46.469332 (+1556838us) ts_tablet_manager.cc:1145] Deleting tablet 577dc789f0cb48f49168124b37a6fba0
0112 20:51:46.469341 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.558474 (+ 89133us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.561163 (+ 2689us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.597884 (+ 36721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1556734}
I20250112 20:51:46.598150 1178 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.598284 1186 tablet_replica.cc:331] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:46.598455 1186 raft_consensus.cc:2238] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.598589 1186 raft_consensus.cc:2267] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.599061 1186 ts_tablet_manager.cc:1905] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.600370 1178 tablet_bootstrap.cc:654] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.606310 1164 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.606495 1164 ts_tablet_manager.cc:1397] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250112 20:51:46.607209 1164 raft_consensus.cc:357] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.607538 1164 raft_consensus.cc:383] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.607623 1164 raft_consensus.cc:738] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.607869 1164 consensus_queue.cc:260] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.608383 1164 ts_tablet_manager.cc:1428] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:46.608649 1164 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.610180 1164 tablet_bootstrap.cc:654] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.610208 2632 raft_consensus.cc:491] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.610332 2632 raft_consensus.cc:513] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.610884 2632 leader_election.cc:290] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.618538 878 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90d6eafea84751b1f1556ca490a4ea (DEFAULT_TABLE table=default.Table_64d69b6c7750480191ca2e8f3234b1e5 [id=bb4b6a35e199408789eaf218773dd769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.625658 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 (DEFAULT_TABLE table=default.Table_03840c2c47d242e6b6459a38152c5afc [id=7e36964613474bd29a4e38314526598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.626183 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.633188 983 leader_election.cc:304] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:46.633466 983 raft_consensus.cc:2681] T 89f9b46da06a476ab9f2a93ca74ff736 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:46.634953 753 tablet_service.cc:1467] Processing CreateTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 (DEFAULT_TABLE table=default.Table_03840c2c47d242e6b6459a38152c5afc [id=7e36964613474bd29a4e38314526598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.636014 892 tablet_service.cc:1467] Processing CreateTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 (DEFAULT_TABLE table=default.Table_03840c2c47d242e6b6459a38152c5afc [id=7e36964613474bd29a4e38314526598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.636374 983 leader_election.cc:304] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.638199 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3db70676f4ac191997d6699056216 (DEFAULT_TABLE table=default.Table_f587cdf889844d2892126c8d93dc87be [id=af836db56b39422ebdac37e648802e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.638674 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b3db70676f4ac191997d6699056216. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:46.642704 847 leader_election.cc:343] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.642822 847 leader_election.cc:304] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.643199 2802 raft_consensus.cc:2743] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.644033 754 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3db70676f4ac191997d6699056216 (DEFAULT_TABLE table=default.Table_f587cdf889844d2892126c8d93dc87be [id=af836db56b39422ebdac37e648802e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.644686 2802 raft_consensus.cc:491] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.644824 2802 raft_consensus.cc:513] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.645392 2802 leader_election.cc:290] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.646173 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.646867 847 leader_election.cc:343] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.647348 847 leader_election.cc:304] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.647883 2802 raft_consensus.cc:2798] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.647971 2802 raft_consensus.cc:491] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.648046 2802 raft_consensus.cc:3054] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.649418 883 tablet_service.cc:1467] Processing CreateTablet for tablet 00b3db70676f4ac191997d6699056216 (DEFAULT_TABLE table=default.Table_f587cdf889844d2892126c8d93dc87be [id=af836db56b39422ebdac37e648802e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.650497 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37fe9e6bcd437ab9317bd69ee27218 (DEFAULT_TABLE table=default.Table_5d0474b9315542a68d09ba65bf803253 [id=ffb5785076b4452aa777855228da9080]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.650910 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c37fe9e6bcd437ab9317bd69ee27218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.652994 2632 raft_consensus.cc:491] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.653128 2632 raft_consensus.cc:513] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.653695 2632 leader_election.cc:290] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.657734 908 raft_consensus.cc:2463] T 8056e2e94fa94fc0b7f03fede018dd8e P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:46.658097 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c897c943cb4fb2aa4d887054f56d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.658509 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90d6eafea84751b1f1556ca490a4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.658820 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.659188 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b3db70676f4ac191997d6699056216. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.660279 983 leader_election.cc:304] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:46.660638 2563 raft_consensus.cc:2798] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:46.660760 2563 raft_consensus.cc:695] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:46.661087 2563 consensus_queue.cc:237] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.663524 2802 raft_consensus.cc:513] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.666003 2802 leader_election.cc:290] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.672919 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:46.673743 847 leader_election.cc:343] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.676095 2802 consensus_queue.cc:1035] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.677904 754 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37fe9e6bcd437ab9317bd69ee27218 (DEFAULT_TABLE table=default.Table_5d0474b9315542a68d09ba65bf803253 [id=ffb5785076b4452aa777855228da9080]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:46.680409 847 leader_election.cc:343] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.680593 847 leader_election.cc:304] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.681085 2802 raft_consensus.cc:2743] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.681852 2787 raft_consensus.cc:2743] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.682024 885 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37fe9e6bcd437ab9317bd69ee27218 (DEFAULT_TABLE table=default.Table_5d0474b9315542a68d09ba65bf803253 [id=ffb5785076b4452aa777855228da9080]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.683629 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3957cc42e3bb439489c93bedb628a0f1 (DEFAULT_TABLE table=default.Table_ff262f32148c4bb9acd0b6492fe0b8e4 [id=d754ff1ceff94dfc9a4418230c5dbe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.683686 2802 consensus_queue.cc:1035] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.684114 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3957cc42e3bb439489c93bedb628a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:46.684551 847 leader_election.cc:343] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.684641 847 leader_election.cc:304] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:46.684978 2802 raft_consensus.cc:2743] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:46.685628 847 leader_election.cc:304] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.685664 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3957cc42e3bb439489c93bedb628a0f1 (DEFAULT_TABLE table=default.Table_ff262f32148c4bb9acd0b6492fe0b8e4 [id=d754ff1ceff94dfc9a4418230c5dbe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.685945 2802 raft_consensus.cc:2798] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.686002 2802 raft_consensus.cc:491] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.686064 2802 raft_consensus.cc:3054] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.689005 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3957cc42e3bb439489c93bedb628a0f1 (DEFAULT_TABLE table=default.Table_ff262f32148c4bb9acd0b6492fe0b8e4 [id=d754ff1ceff94dfc9a4418230c5dbe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.694469 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 02204076f2f44e369a74fe14680538b3 (DEFAULT_TABLE table=default.Table_22606a15a61946b08eec7776233747ed [id=4c44ab48307447de9f54920a583a3137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.694808 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02204076f2f44e369a74fe14680538b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.695256 745 tablet_service.cc:1467] Processing CreateTablet for tablet 02204076f2f44e369a74fe14680538b3 (DEFAULT_TABLE table=default.Table_22606a15a61946b08eec7776233747ed [id=4c44ab48307447de9f54920a583a3137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.696228 875 tablet_service.cc:1467] Processing CreateTablet for tablet 02204076f2f44e369a74fe14680538b3 (DEFAULT_TABLE table=default.Table_22606a15a61946b08eec7776233747ed [id=4c44ab48307447de9f54920a583a3137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.698501 2521 raft_consensus.cc:491] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.698621 2521 raft_consensus.cc:513] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.699203 2521 leader_election.cc:290] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.699879 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.699894 2563 raft_consensus.cc:491] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.700186 2632 consensus_queue.cc:1035] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.700340 2563 raft_consensus.cc:513] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.700953 2563 leader_election.cc:290] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.701320 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 26de001d576047588f5706585a731beb (DEFAULT_TABLE table=default.Table_770aef6ded7a457fb8248f61e8c3a65c [id=f46b33673c914272be51267cb367db11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.701663 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26de001d576047588f5706585a731beb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:46.702073 983 leader_election.cc:343] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 3343e1d7f1d04a498bd0ee00311b6a61
W20250112 20:51:46.702314 847 leader_election.cc:343] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.706830 2108 raft_consensus.cc:491] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.706940 2108 raft_consensus.cc:513] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.707399 2108 leader_election.cc:290] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.707752 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.708278 847 leader_election.cc:343] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.708632 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.708778 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.709012 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 6158b97ccf0a42fdb1a510c5d1e80213 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.709970 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.710203 756 tablet_service.cc:1467] Processing CreateTablet for tablet 26de001d576047588f5706585a731beb (DEFAULT_TABLE table=default.Table_770aef6ded7a457fb8248f61e8c3a65c [id=f46b33673c914272be51267cb367db11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.710489 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.710675 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de03e3114d145aeb81590dbae2c548e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:46.710711 882 tablet_service.cc:1467] Processing CreateTablet for tablet 26de001d576047588f5706585a731beb (DEFAULT_TABLE table=default.Table_770aef6ded7a457fb8248f61e8c3a65c [id=f46b33673c914272be51267cb367db11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.710819 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de03e3114d145aeb81590dbae2c548e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:46.711220 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229e4ef771b4032a4bfae3190a851e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.711560 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de03e3114d145aeb81590dbae2c548e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.700747 710 leader_election.cc:304] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:46.717721 2632 raft_consensus.cc:2798] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.717801 2632 raft_consensus.cc:491] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.717857 2632 raft_consensus.cc:3054] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.734760 1164 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.735023 1164 ts_tablet_manager.cc:1397] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.126s user 0.010s sys 0.022s
I20250112 20:51:46.736145 1164 raft_consensus.cc:357] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.736531 1164 raft_consensus.cc:383] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.736637 1164 raft_consensus.cc:738] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.737035 1164 consensus_queue.cc:260] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.745106 1164 ts_tablet_manager.cc:1428] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:46.751461 1164 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:46.754545 1164 tablet_bootstrap.cc:654] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.755316 2059 consensus_queue.cc:1035] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.756959 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:46.760314 710 leader_election.cc:304] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:46.760826 2059 raft_consensus.cc:2798] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:46.760901 2059 raft_consensus.cc:491] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:46.760973 2059 raft_consensus.cc:3054] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.761584 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02c897c943cb4fb2aa4d887054f56d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.762106 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b90d6eafea84751b1f1556ca490a4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.762530 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.762907 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b3db70676f4ac191997d6699056216. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.763293 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c37fe9e6bcd437ab9317bd69ee27218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.763700 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3957cc42e3bb439489c93bedb628a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.764127 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02204076f2f44e369a74fe14680538b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.764559 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26de001d576047588f5706585a731beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.772383 1185 ts_tablet_manager.cc:1918] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.772505 1185 log.cc:1198] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/921ae136d5e2438e950b96f156f23da3
I20250112 20:51:46.772913 1185 ts_tablet_manager.cc:1939] T 921ae136d5e2438e950b96f156f23da3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:46.774024 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 864) took 1372 ms. Trace:
I20250112 20:51:46.774111 2447 rpcz_store.cc:276] 0112 20:51:45.401593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.408856 (+ 7263us) service_pool.cc:225] Handling call
0112 20:51:46.360768 (+951912us) ts_tablet_manager.cc:1145] Deleting tablet 89f9b46da06a476ab9f2a93ca74ff736
0112 20:51:46.360777 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.463008 (+102231us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.571092 (+108084us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.774005 (+202913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":951728}
I20250112 20:51:46.774457 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 865) took 1349 ms. Trace:
I20250112 20:51:46.774518 2447 rpcz_store.cc:276] 0112 20:51:45.424503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.424549 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:46.774339 (+1349790us) ts_tablet_manager.cc:1145] Deleting tablet 015f54bdae6b4333963f9e085c5189e8
0112 20:51:46.774348 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.774443 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349628}
I20250112 20:51:46.774784 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 866) took 1341 ms. Trace:
I20250112 20:51:46.774837 2447 rpcz_store.cc:276] 0112 20:51:45.432981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.433111 (+ 130us) service_pool.cc:225] Handling call
0112 20:51:46.774695 (+1341584us) ts_tablet_manager.cc:1145] Deleting tablet 0d072ebae78e40c48e158eb65775476c
0112 20:51:46.774702 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.774774 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341369}
I20250112 20:51:46.775089 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 867) took 1333 ms. Trace:
I20250112 20:51:46.775152 2447 rpcz_store.cc:276] 0112 20:51:45.441374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.441417 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:46.774999 (+1333582us) ts_tablet_manager.cc:1145] Deleting tablet 3f6ad2ac50c745448031bdf94e0eea59
0112 20:51:46.775005 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.775066 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333443}
I20250112 20:51:46.775457 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 868) took 1324 ms. Trace:
I20250112 20:51:46.775523 2447 rpcz_store.cc:276] 0112 20:51:45.451339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.452393 (+ 1054us) service_pool.cc:225] Handling call
0112 20:51:46.775335 (+1322942us) ts_tablet_manager.cc:1145] Deleting tablet 69012f97aac1441095aa3bb1548de9cd
0112 20:51:46.775342 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.775443 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322781}
I20250112 20:51:46.775792 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 869) took 1300 ms. Trace:
I20250112 20:51:46.775848 2447 rpcz_store.cc:276] 0112 20:51:45.475754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.475804 (+ 50us) service_pool.cc:225] Handling call
0112 20:51:46.775704 (+1299900us) ts_tablet_manager.cc:1145] Deleting tablet c2664a2e98f74f73ab2dc0adaa2335d3
0112 20:51:46.775711 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.775780 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299726}
I20250112 20:51:46.776119 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 870) took 1292 ms. Trace:
I20250112 20:51:46.776175 2447 rpcz_store.cc:276] 0112 20:51:45.483344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.487108 (+ 3764us) service_pool.cc:225] Handling call
0112 20:51:46.776017 (+1288909us) ts_tablet_manager.cc:1145] Deleting tablet 3b0786594542416cb67c4112921943b3
0112 20:51:46.776024 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.776108 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288694}
I20250112 20:51:46.776376 2447 tablet_replica.cc:331] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:46.776535 2447 raft_consensus.cc:2238] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:46.776674 2447 raft_consensus.cc:2267] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:46.777426 2632 raft_consensus.cc:513] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.778095 2632 leader_election.cc:290] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.781260 2447 ts_tablet_manager.cc:1905] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:46.787240 2802 raft_consensus.cc:513] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.787519 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3957cc42e3bb439489c93bedb628a0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.788024 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02204076f2f44e369a74fe14680538b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.788054 2802 leader_election.cc:290] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.788470 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26de001d576047588f5706585a731beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.798924 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f7f491d69a704f32899e231503da02d0 (DEFAULT_TABLE table=default.Table_acdb4e459bbd4e91bc58571fee9062f3 [id=2a7b8124a3cb4c4a92010971a6bff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.799136 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c37fe9e6bcd437ab9317bd69ee27218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.802919 1163 tablet_bootstrap.cc:492] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.803083 1163 ts_tablet_manager.cc:1397] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.215s user 0.028s sys 0.040s
I20250112 20:51:46.803843 1163 raft_consensus.cc:357] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.804037 1163 raft_consensus.cc:383] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.804170 1163 raft_consensus.cc:738] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.804358 1163 consensus_queue.cc:260] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.804836 1163 ts_tablet_manager.cc:1428] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:46.805107 1163 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.806448 1163 tablet_bootstrap.cc:654] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.811578 2632 raft_consensus.cc:491] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.811695 2632 raft_consensus.cc:513] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.812269 2632 leader_election.cc:290] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.812497 2563 raft_consensus.cc:491] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.812623 2563 raft_consensus.cc:513] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.813239 2563 leader_election.cc:290] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.816160 1163 tablet_bootstrap.cc:492] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.816270 1163 ts_tablet_manager.cc:1397] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20250112 20:51:46.816886 1163 raft_consensus.cc:357] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.817052 1163 raft_consensus.cc:383] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.817093 1163 raft_consensus.cc:738] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.817346 1163 consensus_queue.cc:260] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.817831 1163 ts_tablet_manager.cc:1428] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.818069 1163 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.819181 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.820037 1164 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:46.820187 1164 ts_tablet_manager.cc:1397] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.008s
I20250112 20:51:46.821166 1164 raft_consensus.cc:357] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.821410 1164 raft_consensus.cc:383] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.821480 1164 raft_consensus.cc:738] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.821677 1164 consensus_queue.cc:260] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.822261 1163 tablet_bootstrap.cc:654] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.822263 1164 ts_tablet_manager.cc:1428] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:46.823084 1178 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.823277 1178 ts_tablet_manager.cc:1397] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.225s user 0.015s sys 0.060s
I20250112 20:51:46.824206 1178 raft_consensus.cc:357] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.824468 1178 raft_consensus.cc:383] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.824539 1178 raft_consensus.cc:738] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.824744 1178 consensus_queue.cc:260] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.824918 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:46.825454 1178 ts_tablet_manager.cc:1428] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:46.825779 1178 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.827879 1178 tablet_bootstrap.cc:654] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.834177 1163 tablet_bootstrap.cc:492] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.834333 1163 ts_tablet_manager.cc:1397] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20250112 20:51:46.835196 1163 raft_consensus.cc:357] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.835438 1163 raft_consensus.cc:383] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.835487 1163 raft_consensus.cc:738] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.836807 1178 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.837324 1178 ts_tablet_manager.cc:1397] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250112 20:51:46.837328 1163 consensus_queue.cc:260] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.837837 1163 ts_tablet_manager.cc:1428] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:46.838092 1163 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.838249 1178 raft_consensus.cc:357] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.838551 1178 raft_consensus.cc:383] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.838984 1178 raft_consensus.cc:738] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.839361 1178 consensus_queue.cc:260] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.840312 1178 ts_tablet_manager.cc:1428] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:46.840844 1178 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.842543 1178 tablet_bootstrap.cc:654] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.844991 2108 consensus_queue.cc:1035] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:46.845228 847 leader_election.cc:343] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.855018 1047 raft_consensus.cc:3054] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.858428 2563 raft_consensus.cc:491] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.858552 2563 raft_consensus.cc:513] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.859146 2563 leader_election.cc:290] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
W20250112 20:51:46.860152 983 leader_election.cc:343] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:46.860659 2059 raft_consensus.cc:513] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.861287 2059 leader_election.cc:290] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:46.862720 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 100860us
W20250112 20:51:46.863054 710 leader_election.cc:343] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.863533 2632 consensus_queue.cc:1035] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:46.865461 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f491d69a704f32899e231503da02d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.866577 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8649f8404240df8b750f431a835516 (DEFAULT_TABLE table=default.Table_febe7e80fab0422da18aa05a5609d2f0 [id=e608a8c317bd46889c0cb01d8f67acc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.866897 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8649f8404240df8b750f431a835516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.868544 890 tablet_service.cc:1467] Processing CreateTablet for tablet f7f491d69a704f32899e231503da02d0 (DEFAULT_TABLE table=default.Table_acdb4e459bbd4e91bc58571fee9062f3 [id=2a7b8124a3cb4c4a92010971a6bff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.868916 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f491d69a704f32899e231503da02d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.877108 749 tablet_service.cc:1467] Processing CreateTablet for tablet f7f491d69a704f32899e231503da02d0 (DEFAULT_TABLE table=default.Table_acdb4e459bbd4e91bc58571fee9062f3 [id=2a7b8124a3cb4c4a92010971a6bff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.877579 1023 tablet_service.cc:1467] Processing CreateTablet for tablet c5e07f13efff44d1a0a4bb8508c16e12 (DEFAULT_TABLE table=default.Table_e9a8102300fb471f93b5e5ba18c2f532 [id=e52574d9bcc94faa9c1cd360b9240f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.878367 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e07f13efff44d1a0a4bb8508c16e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.881467 1178 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.881659 1178 ts_tablet_manager.cc:1397] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.002s
I20250112 20:51:46.882467 1178 raft_consensus.cc:357] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.882709 1178 raft_consensus.cc:383] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.882800 1178 raft_consensus.cc:738] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.883028 1178 consensus_queue.cc:260] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.883867 1178 ts_tablet_manager.cc:1428] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:46.884354 1178 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.885052 1017 tablet_service.cc:1467] Processing CreateTablet for tablet bfeda42a3fdf4b38be3ea8560e820e05 (DEFAULT_TABLE table=default.Table_eab8a830ea0c47409a6ae483e247cbe6 [id=08aea797d6be49dd8b9a242dc441b084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.885398 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfeda42a3fdf4b38be3ea8560e820e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.886049 1178 tablet_bootstrap.cc:654] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.890408 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8649f8404240df8b750f431a835516 (DEFAULT_TABLE table=default.Table_febe7e80fab0422da18aa05a5609d2f0 [id=e608a8c317bd46889c0cb01d8f67acc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.890905 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8649f8404240df8b750f431a835516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.891698 881 tablet_service.cc:1467] Processing CreateTablet for tablet c5e07f13efff44d1a0a4bb8508c16e12 (DEFAULT_TABLE table=default.Table_e9a8102300fb471f93b5e5ba18c2f532 [id=e52574d9bcc94faa9c1cd360b9240f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.892158 1163 tablet_bootstrap.cc:654] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.892272 1178 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.892439 1178 ts_tablet_manager.cc:1397] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20250112 20:51:46.893240 1178 raft_consensus.cc:357] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.893494 1178 raft_consensus.cc:383] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.893499 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8649f8404240df8b750f431a835516 (DEFAULT_TABLE table=default.Table_febe7e80fab0422da18aa05a5609d2f0 [id=e608a8c317bd46889c0cb01d8f67acc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.893597 1178 raft_consensus.cc:738] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.893683 738 tablet_service.cc:1467] Processing CreateTablet for tablet bfeda42a3fdf4b38be3ea8560e820e05 (DEFAULT_TABLE table=default.Table_eab8a830ea0c47409a6ae483e247cbe6 [id=08aea797d6be49dd8b9a242dc441b084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.893883 744 tablet_service.cc:1467] Processing CreateTablet for tablet c5e07f13efff44d1a0a4bb8508c16e12 (DEFAULT_TABLE table=default.Table_e9a8102300fb471f93b5e5ba18c2f532 [id=e52574d9bcc94faa9c1cd360b9240f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.893882 1178 consensus_queue.cc:260] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.894737 1178 ts_tablet_manager.cc:1428] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:46.895155 1178 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.895816 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ed7885fed610416792ba4c6df2a80f12 (DEFAULT_TABLE table=default.Table_fdbdfd9f44f6454c9839477f1f437558 [id=83349396b7c14946812ff9284af2f1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.896135 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7885fed610416792ba4c6df2a80f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.896688 1178 tablet_bootstrap.cc:654] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.897150 739 tablet_service.cc:1467] Processing CreateTablet for tablet ed7885fed610416792ba4c6df2a80f12 (DEFAULT_TABLE table=default.Table_fdbdfd9f44f6454c9839477f1f437558 [id=83349396b7c14946812ff9284af2f1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.900328 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 577dc789f0cb48f49168124b37a6fba0 (table default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]) successfully deleted
I20250112 20:51:46.901369 1163 tablet_bootstrap.cc:492] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.901589 1163 ts_tablet_manager.cc:1397] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.001s
I20250112 20:51:46.902427 1163 raft_consensus.cc:357] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.902652 1163 raft_consensus.cc:383] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.902711 1163 raft_consensus.cc:738] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.902961 1163 consensus_queue.cc:260] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.903612 1163 ts_tablet_manager.cc:1428] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.903935 1163 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.904938 882 tablet_service.cc:1467] Processing CreateTablet for tablet bfeda42a3fdf4b38be3ea8560e820e05 (DEFAULT_TABLE table=default.Table_eab8a830ea0c47409a6ae483e247cbe6 [id=08aea797d6be49dd8b9a242dc441b084]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:46.910859 1163 tablet_bootstrap.cc:654] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.914171 1178 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.914338 1178 ts_tablet_manager.cc:1397] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20250112 20:51:46.915112 1178 raft_consensus.cc:357] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.915380 1178 raft_consensus.cc:383] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.915483 1178 raft_consensus.cc:738] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.915732 1163 tablet_bootstrap.cc:492] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:46.915854 1163 ts_tablet_manager.cc:1397] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20250112 20:51:46.915733 1178 consensus_queue.cc:260] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.917224 1178 ts_tablet_manager.cc:1428] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:46.917447 2563 raft_consensus.cc:491] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.917313 1163 raft_consensus.cc:357] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.917613 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:46.917562 2563 raft_consensus.cc:513] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:46.917659 1163 raft_consensus.cc:383] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.917721 1163 raft_consensus.cc:738] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.917910 1178 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.918138 2563 leader_election.cc:290] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:46.918670 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.918735 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.918857 908 raft_consensus.cc:2463] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:46.919435 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.919997 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.920195 908 raft_consensus.cc:2463] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:46.920639 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:46.920763 1178 tablet_bootstrap.cc:654] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.921281 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.917896 1163 consensus_queue.cc:260] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.921940 1163 ts_tablet_manager.cc:1428] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:46.922276 1163 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:46.924350 1163 tablet_bootstrap.cc:654] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:46.924350 447 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:46.925204 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_d4e052ce75a047f98d55947e6cb75b67 [id=467b97b4e4694b2f8551e237429bd0d3] alter complete (version 1)
I20250112 20:51:46.933218 2059 raft_consensus.cc:491] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.933357 2059 raft_consensus.cc:513] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.933903 2059 leader_election.cc:290] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.939146 2521 raft_consensus.cc:491] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.939281 2521 raft_consensus.cc:513] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.939882 2521 leader_election.cc:290] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.940656 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.941313 847 leader_election.cc:343] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.901127 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3343e1d7f1d04a498bd0ee00311b6a61 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
W20250112 20:51:46.947046 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 45918.9us
W20250112 20:51:46.947356 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 54ab0ae10f09469fa4f913e049002481 because the tablet was not found. No further retry: Not found: Tablet not found: 54ab0ae10f09469fa4f913e049002481
W20250112 20:51:46.947553 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet efa7f6a1e9804e6ab822f9d4c004b1b3 because the tablet was not found. No further retry: Not found: Tablet not found: efa7f6a1e9804e6ab822f9d4c004b1b3
I20250112 20:51:46.948105 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 9705feb15f384b5185935eacc521d0e0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:46.948400 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 0bb9c2e6f37f4184a106aed05bae7924 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:46.949299 2563 raft_consensus.cc:491] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.949430 2563 raft_consensus.cc:513] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.950042 2563 leader_election.cc:290] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.950604 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b86e02c3663476d9f4b850bf5982df8" } modify_external_catalogs: true
I20250112 20:51:46.950629 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.950781 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b86e02c3663476d9f4b850bf5982df8" } modify_external_catalogs: true
I20250112 20:51:46.951464 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_03840c2c47d242e6b6459a38152c5afc" } modify_external_catalogs: true
I20250112 20:51:46.951614 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_03840c2c47d242e6b6459a38152c5afc" } modify_external_catalogs: true
I20250112 20:51:46.957531 2521 raft_consensus.cc:491] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.957688 2521 raft_consensus.cc:513] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.957813 1186 ts_tablet_manager.cc:1918] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.957921 1186 log.cc:1198] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/921ae136d5e2438e950b96f156f23da3
I20250112 20:51:46.958312 1186 ts_tablet_manager.cc:1939] T 921ae136d5e2438e950b96f156f23da3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:46.958344 2521 leader_election.cc:290] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.958765 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e07f13efff44d1a0a4bb8508c16e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.958705 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.959198 847 leader_election.cc:343] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.961342 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ee6cb2e8069b41899759a8b3c283152f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:46.962018 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee6cb2e8069b41899759a8b3c283152f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:46.962496 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:46.962677 912 raft_consensus.cc:3054] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.963936 2447 ts_tablet_manager.cc:1918] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:46.964054 2447 log.cc:1198] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/01367a11dea04b33a9255508501b9060
I20250112 20:51:46.964396 2447 ts_tablet_manager.cc:1939] T 01367a11dea04b33a9255508501b9060 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:46.964826 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfeda42a3fdf4b38be3ea8560e820e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:46.970955 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:46.972358 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:46.972579 911 raft_consensus.cc:2463] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:46.973316 1042 raft_consensus.cc:3054] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.974872 2863 tablet.cc:1721] T b930069b6f1b49439dbc0a7abd3fba72 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:46.975761 2903 tablet.cc:1721] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:46.977035 1041 raft_consensus.cc:3054] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.977483 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:46.980862 1178 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:46.981043 1178 ts_tablet_manager.cc:1397] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.023s
I20250112 20:51:46.981820 1178 raft_consensus.cc:357] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.981994 1178 raft_consensus.cc:383] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:46.982056 1178 raft_consensus.cc:738] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:46.982265 1178 consensus_queue.cc:260] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.982770 1178 ts_tablet_manager.cc:1428] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:46.982848 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:46.983036 911 raft_consensus.cc:3054] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:46.983170 1178 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:46.987385 2802 raft_consensus.cc:491] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.987560 2802 raft_consensus.cc:513] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.988351 2802 leader_election.cc:290] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.990862 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:46.991554 847 leader_election.cc:343] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:46.993152 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_bdcb34e5580647dba61719814e3ce733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:46.993631 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:46.994235 2748 raft_consensus.cc:491] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.994563 2748 raft_consensus.cc:513] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:46.995553 2748 leader_election.cc:290] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:46.996405 2748 raft_consensus.cc:491] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.996640 2748 raft_consensus.cc:513] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:46.997529 2748 leader_election.cc:290] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:46.998817 2748 raft_consensus.cc:491] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:46.999418 2748 raft_consensus.cc:513] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.000826 2748 leader_election.cc:290] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.000989 1178 tablet_bootstrap.cc:654] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.001778 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_dd1d3864738a483d917f02e276a04376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.003209 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccb593413124921af94d0b855789b52" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.003872 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b930069b6f1b49439dbc0a7abd3fba72" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.004092 776 raft_consensus.cc:2388] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.004607 1163 tablet_bootstrap.cc:492] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.004921 1163 ts_tablet_manager.cc:1397] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.020s
I20250112 20:51:47.005645 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8056e2e94fa94fc0b7f03fede018dd8e" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.005721 1163 raft_consensus.cc:357] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.005842 773 raft_consensus.cc:3054] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.005898 1163 raft_consensus.cc:383] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.005939 1163 raft_consensus.cc:738] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.006145 1163 consensus_queue.cc:260] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.006687 1163 ts_tablet_manager.cc:1428] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:47.006906 1163 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:47.007534 2757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:47.007959 2757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:47.008090 2757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 20:51:47.008451 1163 tablet_bootstrap.cc:654] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.008610 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8649f8404240df8b750f431a835516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.009387 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 891) took 2046 ms. Trace:
I20250112 20:51:47.009477 1185 rpcz_store.cc:276] 0112 20:51:44.963072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963509 (+ 437us) service_pool.cc:225] Handling call
0112 20:51:46.564402 (+1600893us) ts_tablet_manager.cc:1145] Deleting tablet 921ae136d5e2438e950b96f156f23da3
0112 20:51:46.564409 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.657696 (+ 93287us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.772257 (+114561us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.009336 (+237079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600732}
I20250112 20:51:47.009817 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 896) took 2046 ms. Trace:
I20250112 20:51:47.009909 1185 rpcz_store.cc:276] 0112 20:51:44.963512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963603 (+ 91us) service_pool.cc:225] Handling call
0112 20:51:47.009719 (+2046116us) ts_tablet_manager.cc:1145] Deleting tablet d706f88f14ab4fd9a459944863afaacd
0112 20:51:47.009728 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.009782 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2046006}
I20250112 20:51:47.010205 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 888) took 2047 ms. Trace:
I20250112 20:51:47.010290 1185 rpcz_store.cc:276] 0112 20:51:44.962743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.963941 (+ 1198us) service_pool.cc:225] Handling call
0112 20:51:47.010100 (+2046159us) ts_tablet_manager.cc:1145] Deleting tablet 9856e6b625d04f17b0869cee08f69142
0112 20:51:47.010107 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.010168 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045962}
I20250112 20:51:47.010521 1185 tablet_replica.cc:331] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:47.010739 1185 raft_consensus.cc:2238] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.010883 1185 raft_consensus.cc:2267] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.011415 1185 ts_tablet_manager.cc:1905] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.016258 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e07f13efff44d1a0a4bb8508c16e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.017693 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfeda42a3fdf4b38be3ea8560e820e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.018121 2802 raft_consensus.cc:491] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.018261 2802 raft_consensus.cc:513] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.018842 2802 leader_election.cc:290] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.019420 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.020344 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d62facea3c274e3ca3bc123e1402be3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.020891 1163 tablet_bootstrap.cc:492] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.021005 1163 ts_tablet_manager.cc:1397] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20250112 20:51:47.021025 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7885fed610416792ba4c6df2a80f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.021660 1163 raft_consensus.cc:357] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.021843 1163 raft_consensus.cc:383] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.021891 1163 raft_consensus.cc:738] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.022099 1163 consensus_queue.cc:260] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.022637 1163 ts_tablet_manager.cc:1428] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:47.022866 1163 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.023022 1164 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.024425 1163 tablet_bootstrap.cc:654] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.024596 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f491d69a704f32899e231503da02d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:47.025130 2757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:47.025267 2757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:47.025326 2757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:47.025393 2757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:47.026539 1163 tablet_bootstrap.cc:492] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.026647 1163 ts_tablet_manager.cc:1397] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:47.026782 1164 tablet_bootstrap.cc:654] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.027307 1163 raft_consensus.cc:357] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.027483 1163 raft_consensus.cc:383] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.027536 1163 raft_consensus.cc:738] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.027719 1163 consensus_queue.cc:260] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.028002 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.028236 776 raft_consensus.cc:2463] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:47.028465 1163 ts_tablet_manager.cc:1428] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.028738 1163 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.030375 1163 tablet_bootstrap.cc:654] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.033572 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6f81ce5122864ba79e643d6de8b6f857 (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.034749 753 tablet_service.cc:1467] Processing CreateTablet for tablet e65c89f6026747dd88f960a638087607 (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.038197 1163 tablet_bootstrap.cc:492] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.038378 1163 ts_tablet_manager.cc:1397] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250112 20:51:47.038743 754 tablet_service.cc:1467] Processing CreateTablet for tablet 91243c9c6deb447e8246f7abe2691f55 (DEFAULT_TABLE table=default.Table_d4025212c78a43f2af864cca35e42251 [id=818a41b6ffa34b2fa69fbe0fb961cb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.039047 1163 raft_consensus.cc:357] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.039300 1163 raft_consensus.cc:383] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.039390 1163 raft_consensus.cc:738] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.039633 1163 consensus_queue.cc:260] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.039912 756 tablet_service.cc:1467] Processing CreateTablet for tablet 417ada5111454c54a85cc95db5974bf9 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.040248 1163 ts_tablet_manager.cc:1428] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:47.040534 1163 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:47.041308 847 leader_election.cc:343] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.042073 1178 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:47.042236 1163 tablet_bootstrap.cc:654] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.042234 1178 ts_tablet_manager.cc:1397] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.003s
I20250112 20:51:47.042690 2757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35199,127.0.104.61:39271,127.0.104.60:45153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=39277
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:47.042798 884 tablet_service.cc:1467] Processing CreateTablet for tablet ed7885fed610416792ba4c6df2a80f12 (DEFAULT_TABLE table=default.Table_fdbdfd9f44f6454c9839477f1f437558 [id=83349396b7c14946812ff9284af2f1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.043025 1178 raft_consensus.cc:357] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.043218 1178 raft_consensus.cc:383] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.043334 1178 raft_consensus.cc:738] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.043578 885 tablet_service.cc:1467] Processing CreateTablet for tablet e65c89f6026747dd88f960a638087607 (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.043555 1178 consensus_queue.cc:260] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.043955 892 tablet_service.cc:1467] Processing CreateTablet for tablet 91243c9c6deb447e8246f7abe2691f55 (DEFAULT_TABLE table=default.Table_d4025212c78a43f2af864cca35e42251 [id=818a41b6ffa34b2fa69fbe0fb961cb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.044117 1178 ts_tablet_manager.cc:1428] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:47.043290 875 tablet_service.cc:1467] Processing CreateTablet for tablet 6f81ce5122864ba79e643d6de8b6f857 (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.045949 2757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:47.046989 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.047237 776 raft_consensus.cc:3054] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.048359 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.048985 2757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:51:47.050041 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5c228264e345348c9a1bf4357e7964 (DEFAULT_TABLE table=default.Table_53a9e79cf0434f96bdffc0b5c7f26934 [id=33a4c64230994126bb509caf6846eda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.050737 1163 tablet_bootstrap.cc:492] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.050954 1163 ts_tablet_manager.cc:1397] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20250112 20:51:47.051748 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 91243c9c6deb447e8246f7abe2691f55 (DEFAULT_TABLE table=default.Table_d4025212c78a43f2af864cca35e42251 [id=818a41b6ffa34b2fa69fbe0fb961cb29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.051733 1163 raft_consensus.cc:357] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.052066 1163 raft_consensus.cc:383] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.052284 1163 raft_consensus.cc:738] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.052279 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 417ada5111454c54a85cc95db5974bf9 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.052279 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5c228264e345348c9a1bf4357e7964 (DEFAULT_TABLE table=default.Table_53a9e79cf0434f96bdffc0b5c7f26934 [id=33a4c64230994126bb509caf6846eda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.052596 1163 consensus_queue.cc:260] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.053000 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 02b78ae4ec214013b290c2b76c7b9965 (DEFAULT_TABLE table=default.Table_ea686bd946c94aeca2db6212d2fa6b7d [id=82fcbc22fb634040a0fcd54e614cba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.053435 1163 ts_tablet_manager.cc:1428] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:47.053743 1163 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.053980 737 tablet_service.cc:1467] Processing CreateTablet for tablet 02264625e0d9441882ecefcc8cc36227 (DEFAULT_TABLE table=default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77 [id=f4c0dc2544874f5da96840fa475c622f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.053648 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 6f81ce5122864ba79e643d6de8b6f857 (DEFAULT_TABLE table=default.Table_7c19fb4385ba42d8a7501cd54c098be9 [id=54e4ef5bcf42476ab9c9d177b239c7d2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:47.054764 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d62facea3c274e3ca3bc123e1402be3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:47.055991 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdcb34e5580647dba61719814e3ce733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.056193 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417ada5111454c54a85cc95db5974bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.056535 1163 tablet_bootstrap.cc:654] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.056653 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91243c9c6deb447e8246f7abe2691f55. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:47.056962 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd1d3864738a483d917f02e276a04376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.057124 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e5c228264e345348c9a1bf4357e7964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.057530 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b78ae4ec214013b290c2b76c7b9965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.058065 1017 tablet_service.cc:1467] Processing CreateTablet for tablet e65c89f6026747dd88f960a638087607 (DEFAULT_TABLE table=default.Table_a113b0bc0435433eb5260a68d20896f8 [id=ba3ae66c4a454ae4adca1b927d1997d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.058401 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65c89f6026747dd88f960a638087607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.059687 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 872) took 1553 ms. Trace:
I20250112 20:51:47.059775 2447 rpcz_store.cc:276] 0112 20:51:45.505878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.506363 (+ 485us) service_pool.cc:225] Handling call
0112 20:51:46.776342 (+1269979us) ts_tablet_manager.cc:1145] Deleting tablet 01367a11dea04b33a9255508501b9060
0112 20:51:46.776350 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.861564 (+ 85214us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.963808 (+102244us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.059669 (+ 95861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269815}
I20250112 20:51:47.060067 2447 tablet_replica.cc:331] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:47.060267 2447 raft_consensus.cc:2238] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.060432 2447 raft_consensus.cc:2267] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.060948 2447 ts_tablet_manager.cc:1905] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.064425 1163 tablet_bootstrap.cc:492] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.064539 1163 ts_tablet_manager.cc:1397] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250112 20:51:47.065454 1163 raft_consensus.cc:357] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.065634 1163 raft_consensus.cc:383] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.065677 1163 raft_consensus.cc:738] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.065888 1163 consensus_queue.cc:260] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.066422 1163 ts_tablet_manager.cc:1428] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.066660 1163 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.068238 1163 tablet_bootstrap.cc:654] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.070230 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f81ce5122864ba79e643d6de8b6f857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.070950 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 02264625e0d9441882ecefcc8cc36227 (DEFAULT_TABLE table=default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77 [id=f4c0dc2544874f5da96840fa475c622f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.071281 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02264625e0d9441882ecefcc8cc36227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.074316 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 510a8bcaa36d496f87e45736d6246b84 (DEFAULT_TABLE table=default.Table_7b12d34248404d5485e174ce460e7d56 [id=a03ea95657794ad1ac758c56386604f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.074616 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510a8bcaa36d496f87e45736d6246b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.075471 1163 tablet_bootstrap.cc:492] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.075644 1163 ts_tablet_manager.cc:1397] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250112 20:51:47.076388 1163 raft_consensus.cc:357] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.076632 1163 raft_consensus.cc:383] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.076740 1163 raft_consensus.cc:738] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.077011 1163 consensus_queue.cc:260] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.077584 1163 ts_tablet_manager.cc:1428] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:47.077872 1163 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:47.078003 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d94c67881e954c0ebecf5c31de665ac0 (table default.Table_cb6c84ab3dd344d28800d498d1816591 [id=54ce1bfbecca4b36bc6fa53ec55ec830], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d94c67881e954c0ebecf5c31de665ac0 consensus state has no leader
I20250112 20:51:47.078805 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.079141 741 tablet_service.cc:1467] Processing CreateTablet for tablet 02b78ae4ec214013b290c2b76c7b9965 (DEFAULT_TABLE table=default.Table_ea686bd946c94aeca2db6212d2fa6b7d [id=82fcbc22fb634040a0fcd54e614cba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.079628 1163 tablet_bootstrap.cc:654] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.080112 2903 tablet.cc:1721] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:47.084265 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b7d3364377a4471b9d511e51e1467297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.084821 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7d3364377a4471b9d511e51e1467297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.122366 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 1b228712feb24dd1af8d8f52be35b8c8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.122954 2563 raft_consensus.cc:491] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.123121 2563 raft_consensus.cc:513] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.123507 2787 raft_consensus.cc:491] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.123649 2787 raft_consensus.cc:513] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.123912 2563 leader_election.cc:290] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.124190 2787 leader_election.cc:290] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.130649 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6 (DEFAULT_TABLE table=default.Table_29bb69809f3848deaca3e824da508612 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.131366 2059 raft_consensus.cc:491] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.131526 2059 raft_consensus.cc:513] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.132165 2059 leader_election.cc:290] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.152346 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 89f9b46da06a476ab9f2a93ca74ff736 (table default.Table_a6c9111b483148718ccd2bfb2a54698b [id=b3b49691fa8a41dda2abad8d2f59e423]) successfully deleted
I20250112 20:51:47.160708 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7885fed610416792ba4c6df2a80f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.161952 888 tablet_service.cc:1467] Processing CreateTablet for tablet 02b78ae4ec214013b290c2b76c7b9965 (DEFAULT_TABLE table=default.Table_ea686bd946c94aeca2db6212d2fa6b7d [id=82fcbc22fb634040a0fcd54e614cba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.162698 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5c228264e345348c9a1bf4357e7964 (DEFAULT_TABLE table=default.Table_53a9e79cf0434f96bdffc0b5c7f26934 [id=33a4c64230994126bb509caf6846eda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.163093 877 tablet_service.cc:1467] Processing CreateTablet for tablet 02264625e0d9441882ecefcc8cc36227 (DEFAULT_TABLE table=default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77 [id=f4c0dc2544874f5da96840fa475c622f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.163419 891 tablet_service.cc:1467] Processing CreateTablet for tablet 510a8bcaa36d496f87e45736d6246b84 (DEFAULT_TABLE table=default.Table_7b12d34248404d5485e174ce460e7d56 [id=a03ea95657794ad1ac758c56386604f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.163779 876 tablet_service.cc:1467] Processing CreateTablet for tablet 1b228712feb24dd1af8d8f52be35b8c8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.164111 874 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6 (DEFAULT_TABLE table=default.Table_29bb69809f3848deaca3e824da508612 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.165357 2632 raft_consensus.cc:491] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.167243 2787 raft_consensus.cc:491] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.167366 2787 raft_consensus.cc:513] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.167932 2787 leader_election.cc:290] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.173677 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65c89f6026747dd88f960a638087607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.173882 912 raft_consensus.cc:2463] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.174826 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.175036 907 raft_consensus.cc:2388] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.176630 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.176826 913 raft_consensus.cc:2463] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.177206 878 tablet_service.cc:1467] Processing CreateTablet for tablet 417ada5111454c54a85cc95db5974bf9 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.179792 2787 raft_consensus.cc:491] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.179960 2787 raft_consensus.cc:513] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } }
I20250112 20:51:47.180786 2787 leader_election.cc:290] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585)
I20250112 20:51:47.187064 983 leader_election.cc:304] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.187510 2787 raft_consensus.cc:2798] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.187593 2787 raft_consensus.cc:491] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.187654 2787 raft_consensus.cc:3054] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.189883 1185 ts_tablet_manager.cc:1918] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.190002 1185 log.cc:1198] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e9f669a739d94f99a5b7e9ccccd29545
I20250112 20:51:47.190343 1185 ts_tablet_manager.cc:1939] T e9f669a739d94f99a5b7e9ccccd29545 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:47.197150 983 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:47.197512 2802 raft_consensus.cc:491] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.197652 2802 raft_consensus.cc:513] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.198261 2802 leader_election.cc:290] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.199105 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:47.199628 847 leader_election.cc:343] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.202443 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91243c9c6deb447e8246f7abe2691f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.205188 1178 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:47.205760 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 894) took 2291 ms. Trace:
I20250112 20:51:47.205840 1186 rpcz_store.cc:276] 0112 20:51:44.914095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.914143 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:46.598241 (+1684098us) ts_tablet_manager.cc:1145] Deleting tablet 921ae136d5e2438e950b96f156f23da3
0112 20:51:46.598249 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:46.855217 (+256968us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:46.957686 (+102469us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.205716 (+248030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683954}
I20250112 20:51:47.206177 1186 tablet_replica.cc:331] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:47.206349 1186 raft_consensus.cc:2238] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:47.206611 1186 raft_consensus.cc:2267] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.207123 1186 ts_tablet_manager.cc:1905] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.205194 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f81ce5122864ba79e643d6de8b6f857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.208760 1178 tablet_bootstrap.cc:654] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.209533 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b78ae4ec214013b290c2b76c7b9965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.212141 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.212393 908 raft_consensus.cc:2388] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.213330 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417ada5111454c54a85cc95db5974bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.216794 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b228712feb24dd1af8d8f52be35b8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.219924 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510a8bcaa36d496f87e45736d6246b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.222999 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02264625e0d9441882ecefcc8cc36227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.226209 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.229033 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e5c228264e345348c9a1bf4357e7964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.233687 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f81ce5122864ba79e643d6de8b6f857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.235061 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02264625e0d9441882ecefcc8cc36227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.247874 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1bc274860e8247998efa842371c1f86a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.248876 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4f98a4c818a2481d8f8e64ea64693585" } new_table_name: "default.Table_2b37091fbda64e07b5641542b2961d72" modify_external_catalogs: true
W20250112 20:51:47.250156 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc274860e8247998efa842371c1f86a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.251343 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2784d0f26fb740fe894e7750b460886d" } new_table_name: "default.Table_d308f6e26ddd443c9a3b7c3f69a7bfc1" modify_external_catalogs: true
I20250112 20:51:47.251960 911 raft_consensus.cc:2463] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.256142 2787 raft_consensus.cc:513] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.256886 2787 leader_election.cc:290] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.257563 2787 raft_consensus.cc:491] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.257678 2787 raft_consensus.cc:513] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.258186 2787 leader_election.cc:290] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.258882 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.259078 911 raft_consensus.cc:3054] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.261622 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7561a010074e4f19ba6a480d869bb939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.262290 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7561a010074e4f19ba6a480d869bb939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.264541 2521 raft_consensus.cc:491] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.264662 2521 raft_consensus.cc:513] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.265259 2521 leader_election.cc:290] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.266136 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.267227 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.267467 908 raft_consensus.cc:2463] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:47.267810 847 leader_election.cc:343] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.268636 1178 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:47.268817 1178 ts_tablet_manager.cc:1397] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.007s
I20250112 20:51:47.269582 1178 raft_consensus.cc:357] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.269806 1178 raft_consensus.cc:383] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.269908 1178 raft_consensus.cc:738] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.270123 1178 consensus_queue.cc:260] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.270346 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.270665 1178 ts_tablet_manager.cc:1428] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:47.282111 1163 tablet_bootstrap.cc:492] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.282280 1163 ts_tablet_manager.cc:1397] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.204s user 0.050s sys 0.091s
I20250112 20:51:47.282439 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.282635 1045 raft_consensus.cc:2463] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.283106 1163 raft_consensus.cc:357] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.283330 1163 raft_consensus.cc:383] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.283378 1163 raft_consensus.cc:738] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.283619 1163 consensus_queue.cc:260] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.283973 2563 raft_consensus.cc:491] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.284150 1163 ts_tablet_manager.cc:1428] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:47.284091 2563 raft_consensus.cc:513] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.284400 1163 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.284639 2563 leader_election.cc:290] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.285022 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.285235 1045 raft_consensus.cc:2463] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.286309 1163 tablet_bootstrap.cc:654] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.286309 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557fa6445eee481dbc4076f67fe63466" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:47.287094 983 leader_election.cc:343] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:47.287196 983 leader_election.cc:304] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.287557 983 leader_election.cc:304] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:47.287874 983 leader_election.cc:343] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:47.287953 983 leader_election.cc:304] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.288368 1178 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:47.288547 2563 raft_consensus.cc:2743] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.289265 2787 raft_consensus.cc:2743] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.289536 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.289731 910 raft_consensus.cc:2463] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.289928 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.290452 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557fa6445eee481dbc4076f67fe63466" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.290575 906 raft_consensus.cc:2463] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.293011 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.293236 909 raft_consensus.cc:2463] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.295640 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.295799 1045 raft_consensus.cc:2463] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.296492 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.296712 909 raft_consensus.cc:2463] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.165498 2632 raft_consensus.cc:513] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.290096 907 raft_consensus.cc:2463] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.290169 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.297936 2632 leader_election.cc:290] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.298024 913 raft_consensus.cc:2463] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.301855 773 raft_consensus.cc:2463] T 8056e2e94fa94fc0b7f03fede018dd8e P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.303326 1163 tablet_bootstrap.cc:492] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.303478 1163 ts_tablet_manager.cc:1397] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20250112 20:51:47.304373 1163 raft_consensus.cc:357] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.304570 1163 raft_consensus.cc:383] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.304628 1163 raft_consensus.cc:738] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.304898 1163 consensus_queue.cc:260] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.305581 1163 ts_tablet_manager.cc:1428] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.305897 1163 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.307556 983 leader_election.cc:304] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.307884 1163 tablet_bootstrap.cc:654] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.309141 2926 raft_consensus.cc:2743] T b930069b6f1b49439dbc0a7abd3fba72 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.309675 1178 tablet_bootstrap.cc:654] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.309732 738 tablet_service.cc:1467] Processing CreateTablet for tablet 1b228712feb24dd1af8d8f52be35b8c8 (DEFAULT_TABLE table=default.Table_71108c2ed35249678274682b502f927a [id=203f849461b04c53af7e83c1af071e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.309391 983 leader_election.cc:304] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.310029 983 leader_election.cc:304] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.310127 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b228712feb24dd1af8d8f52be35b8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.310252 2926 raft_consensus.cc:2798] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.310308 2926 raft_consensus.cc:491] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.310377 2926 raft_consensus.cc:3054] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.315999 2787 raft_consensus.cc:2798] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.316120 2787 raft_consensus.cc:491] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.316196 2787 raft_consensus.cc:3054] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.318099 1163 tablet_bootstrap.cc:492] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.318249 1163 ts_tablet_manager.cc:1397] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20250112 20:51:47.319139 1163 raft_consensus.cc:357] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.319368 1163 raft_consensus.cc:383] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.319424 1163 raft_consensus.cc:738] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.321620 1163 consensus_queue.cc:260] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.322208 1163 ts_tablet_manager.cc:1428] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:47.322518 1163 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.324659 1163 tablet_bootstrap.cc:654] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.327716 847 leader_election.cc:304] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.328308 2802 raft_consensus.cc:2798] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.328403 2802 raft_consensus.cc:491] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.328480 2802 raft_consensus.cc:3054] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:47.329602 847 leader_election.cc:343] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.329802 847 leader_election.cc:304] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.330103 2521 raft_consensus.cc:2743] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:47.330557 847 leader_election.cc:343] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.333163 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417ada5111454c54a85cc95db5974bf9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:47.334376 2932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:47.335073 2933 raft_consensus.cc:491] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.335361 2933 raft_consensus.cc:513] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.336724 2933 leader_election.cc:290] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.336771 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:47.339934 2929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:47.330726 847 leader_election.cc:304] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.342125 2108 raft_consensus.cc:2743] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.345516 2563 raft_consensus.cc:2743] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.346356 1163 tablet_bootstrap.cc:492] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.346520 1163 ts_tablet_manager.cc:1397] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250112 20:51:47.347345 1163 raft_consensus.cc:357] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.347544 1163 raft_consensus.cc:383] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.347625 1163 raft_consensus.cc:738] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.347868 1163 consensus_queue.cc:260] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.348395 1163 ts_tablet_manager.cc:1428] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:47.348698 1163 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.350656 1178 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:47.350797 1178 ts_tablet_manager.cc:1397] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.000s
I20250112 20:51:47.351226 2447 ts_tablet_manager.cc:1918] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.351343 2447 log.cc:1198] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e9f669a739d94f99a5b7e9ccccd29545
I20250112 20:51:47.351758 2447 ts_tablet_manager.cc:1939] T e9f669a739d94f99a5b7e9ccccd29545 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:47.351926 1178 raft_consensus.cc:357] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.352120 1178 raft_consensus.cc:383] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.352176 1178 raft_consensus.cc:738] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.352380 1178 consensus_queue.cc:260] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.301767 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.356976 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b228712feb24dd1af8d8f52be35b8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.353106 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.357939 775 raft_consensus.cc:3054] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.358692 1163 tablet_bootstrap.cc:654] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.301651 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.360375 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6 (DEFAULT_TABLE table=default.Table_29bb69809f3848deaca3e824da508612 [id=b56651754d344c37913ce7942cd314fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.360924 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.360895 776 raft_consensus.cc:2463] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:47.361284 911 raft_consensus.cc:2463] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.361843 983 leader_election.cc:304] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.363308 1178 ts_tablet_manager.cc:1428] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:47.364760 2563 raft_consensus.cc:2798] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.364959 2563 raft_consensus.cc:695] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.365296 2563 consensus_queue.cc:237] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.367908 2926 raft_consensus.cc:513] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.369550 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.369727 911 raft_consensus.cc:3054] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.369616 2926 leader_election.cc:290] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.373598 1178 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:47.376086 1163 tablet_bootstrap.cc:492] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.376219 1163 ts_tablet_manager.cc:1397] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20250112 20:51:47.376443 1178 tablet_bootstrap.cc:654] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.373309 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 894) took 2409 ms. Trace:
I20250112 20:51:47.377059 1163 raft_consensus.cc:357] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.377266 1163 raft_consensus.cc:383] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.377337 1163 raft_consensus.cc:738] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.377664 1163 consensus_queue.cc:260] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.378201 1163 ts_tablet_manager.cc:1428] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.378476 1163 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.380064 1163 tablet_bootstrap.cc:654] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:47.383476 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39271, user_credentials={real_user=slave}} blocked reactor thread for 117826us
W20250112 20:51:47.385299 983 consensus_peers.cc:487] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 577dc789f0cb48f49168124b37a6fba0. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:47.387476 1163 tablet_bootstrap.cc:492] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.387629 1163 ts_tablet_manager.cc:1397] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250112 20:51:47.388415 1163 raft_consensus.cc:357] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.388641 1163 raft_consensus.cc:383] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.388723 1163 raft_consensus.cc:738] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.389050 1163 consensus_queue.cc:260] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.389837 1163 ts_tablet_manager.cc:1428] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 20:51:47.390048 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 238625us
I20250112 20:51:47.390151 1163 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.392125 1163 tablet_bootstrap.cc:654] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:47.396872 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 015f54bdae6b4333963f9e085c5189e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015f54bdae6b4333963f9e085c5189e8 already in progress: creating tablet
W20250112 20:51:47.397123 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 0d072ebae78e40c48e158eb65775476c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d072ebae78e40c48e158eb65775476c already in progress: creating tablet
W20250112 20:51:47.397323 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3f6ad2ac50c745448031bdf94e0eea59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f6ad2ac50c745448031bdf94e0eea59 already in progress: creating tablet
W20250112 20:51:47.397507 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 69012f97aac1441095aa3bb1548de9cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69012f97aac1441095aa3bb1548de9cd already in progress: creating tablet
W20250112 20:51:47.397697 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c2664a2e98f74f73ab2dc0adaa2335d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2664a2e98f74f73ab2dc0adaa2335d3 already in progress: creating tablet
W20250112 20:51:47.397871 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3b0786594542416cb67c4112921943b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0786594542416cb67c4112921943b3 already in progress: creating tablet
I20250112 20:51:47.399283 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 01367a11dea04b33a9255508501b9060 (table default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]) successfully deleted
I20250112 20:51:47.400429 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 921ae136d5e2438e950b96f156f23da3 (table default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]) successfully deleted
I20250112 20:51:47.302178 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.302321 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.402624 1185 rpcz_store.cc:276] 0112 20:51:44.963353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.964110 (+ 757us) service_pool.cc:225] Handling call
0112 20:51:47.010458 (+2046348us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:47.010465 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.083381 (+ 72916us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.189753 (+106372us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.373275 (+183522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2046232}
I20250112 20:51:47.404980 1163 tablet_bootstrap.cc:492] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.405140 1163 ts_tablet_manager.cc:1397] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20250112 20:51:47.406234 2926 raft_consensus.cc:491] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.406364 2926 raft_consensus.cc:513] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.406939 2926 leader_election.cc:290] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.407315 1163 raft_consensus.cc:357] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.407490 1163 raft_consensus.cc:383] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.407573 1163 raft_consensus.cc:738] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.407632 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.407804 1163 consensus_queue.cc:260] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.407894 913 raft_consensus.cc:2463] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.408344 983 leader_election.cc:304] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.401396 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91243c9c6deb447e8246f7abe2691f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.408933 2563 raft_consensus.cc:2798] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.409044 2563 raft_consensus.cc:491] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.409134 2563 raft_consensus.cc:3054] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.409488 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14b0bfd973497b97ed6a3c7bec1aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.374207 748 tablet_service.cc:1467] Processing CreateTablet for tablet 510a8bcaa36d496f87e45736d6246b84 (DEFAULT_TABLE table=default.Table_7b12d34248404d5485e174ce460e7d56 [id=a03ea95657794ad1ac758c56386604f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.357756 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.404570 766 raft_consensus.cc:2463] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:47.412302 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510a8bcaa36d496f87e45736d6246b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.357654 757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.413012 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b78ae4ec214013b290c2b76c7b9965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.413033 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 921ae136d5e2438e950b96f156f23da3 (table default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]) successfully deleted
W20250112 20:51:47.413261 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d706f88f14ab4fd9a459944863afaacd because the tablet was not found. No further retry: Not found: Tablet not found: d706f88f14ab4fd9a459944863afaacd
W20250112 20:51:47.413453 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9856e6b625d04f17b0869cee08f69142 because the tablet was not found. No further retry: Not found: Tablet not found: 9856e6b625d04f17b0869cee08f69142
I20250112 20:51:47.415136 2059 raft_consensus.cc:491] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.415266 2059 raft_consensus.cc:513] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.415822 2059 leader_election.cc:290] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:47.416219 710 leader_election.cc:343] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:47.416383 710 leader_election.cc:343] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:47.416500 710 leader_election.cc:343] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.417440 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 889) took 2454 ms. Trace:
I20250112 20:51:47.417526 1185 rpcz_store.cc:276] 0112 20:51:44.962846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.964156 (+ 1310us) service_pool.cc:225] Handling call
0112 20:51:47.417349 (+2453193us) ts_tablet_manager.cc:1145] Deleting tablet 6092a31de9bc47e4bae5f5aebb180c6d
0112 20:51:47.417359 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.417425 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2453071}
I20250112 20:51:47.417804 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65c89f6026747dd88f960a638087607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.420400 2521 raft_consensus.cc:491] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:47.420475 710 leader_election.cc:343] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.420538 2521 raft_consensus.cc:513] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.421126 2521 leader_election.cc:290] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:47.421628 710 leader_election.cc:343] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.421797 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e5c228264e345348c9a1bf4357e7964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.421777 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:47.422411 847 leader_election.cc:343] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.422957 2926 raft_consensus.cc:491] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.423192 2926 raft_consensus.cc:513] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.417822 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 898) took 2441 ms. Trace:
I20250112 20:51:47.424947 1185 rpcz_store.cc:276] 0112 20:51:44.976243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.976290 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:47.417735 (+2441445us) ts_tablet_manager.cc:1145] Deleting tablet 3dc719cd84124082b782016011e37c8b
0112 20:51:47.417743 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.417812 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2441263}
I20250112 20:51:47.425382 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 899) took 2443 ms. Trace:
I20250112 20:51:47.425443 1185 rpcz_store.cc:276] 0112 20:51:44.982102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.982156 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:47.425263 (+2443107us) ts_tablet_manager.cc:1145] Deleting tablet cc9d5d182827465abd3799d3ad9a6484
0112 20:51:47.425276 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.425370 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442951}
I20250112 20:51:47.425710 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.426148 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 900) took 2366 ms. Trace:
I20250112 20:51:47.426234 1185 rpcz_store.cc:276] 0112 20:51:45.060027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.060579 (+ 552us) service_pool.cc:225] Handling call
0112 20:51:47.426062 (+2365483us) ts_tablet_manager.cc:1145] Deleting tablet d80791d8582446b89adf212d3d6507eb
0112 20:51:47.426071 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.426135 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2365295}
I20250112 20:51:47.426319 2926 leader_election.cc:290] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.426600 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 901) took 2344 ms. Trace:
I20250112 20:51:47.426695 1185 rpcz_store.cc:276] 0112 20:51:45.082469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.082523 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:47.426484 (+2343961us) ts_tablet_manager.cc:1145] Deleting tablet e8bea4d83ea644a0930a29f61e7a5fc8
0112 20:51:47.426496 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.426585 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343775}
I20250112 20:51:47.426997 1185 tablet_replica.cc:331] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:47.427246 1185 raft_consensus.cc:2238] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:47.427520 1185 raft_consensus.cc:2267] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.428046 1185 ts_tablet_manager.cc:1905] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:47.429044 710 leader_election.cc:343] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.429206 710 leader_election.cc:304] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.408344 1163 ts_tablet_manager.cc:1428] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:47.429751 1163 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.429935 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e9f669a739d94f99a5b7e9ccccd29545 (table default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]) successfully deleted
W20250112 20:51:47.430181 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6092a31de9bc47e4bae5f5aebb180c6d because the tablet was not found. No further retry: Not found: Tablet not found: 6092a31de9bc47e4bae5f5aebb180c6d
W20250112 20:51:47.431164 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cc9d5d182827465abd3799d3ad9a6484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9d5d182827465abd3799d3ad9a6484 already in progress: creating tablet
W20250112 20:51:47.431354 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d80791d8582446b89adf212d3d6507eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d80791d8582446b89adf212d3d6507eb already in progress: creating tablet
I20250112 20:51:47.431494 2059 raft_consensus.cc:2798] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.431571 2059 raft_consensus.cc:491] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.431627 2059 raft_consensus.cc:3054] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:47.431605 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e8bea4d83ea644a0930a29f61e7a5fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8bea4d83ea644a0930a29f61e7a5fc8 already in progress: creating tablet
I20250112 20:51:47.431950 710 leader_election.cc:304] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:47.434052 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 549d92191a594788bf639cd60a6b5142 consensus state has no leader
I20250112 20:51:47.434221 1164 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.434329 1164 ts_tablet_manager.cc:1397] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.412s user 0.018s sys 0.018s
I20250112 20:51:47.434799 2923 raft_consensus.cc:2798] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.434859 2923 raft_consensus.cc:491] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.434911 2923 raft_consensus.cc:3054] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:47.435199 710 leader_election.cc:343] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:47.435314 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01bad472601a42268bacb63116352f75 has no consensus state
W20250112 20:51:47.435343 710 consensus_peers.cc:487] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:47.436304 710 leader_election.cc:304] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.436684 2927 raft_consensus.cc:2798] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.436738 2927 raft_consensus.cc:491] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.436805 2927 raft_consensus.cc:3054] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.437793 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53551931b23d43da89e9193cdf034a63" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.437987 765 raft_consensus.cc:2388] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.438258 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f143227e79844a7393b323222bf7eaef" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.438413 1163 tablet_bootstrap.cc:654] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.439632 1164 raft_consensus.cc:357] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.439836 1164 raft_consensus.cc:383] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.439918 1164 raft_consensus.cc:738] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.440145 1164 consensus_queue.cc:260] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.440618 1164 ts_tablet_manager.cc:1428] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250112 20:51:47.440900 1164 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.442122 1164 tablet_bootstrap.cc:654] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.443177 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc90d3722fd14d59b9994655cd30ef2b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.443248 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161951bfabff485dbf7dceeac2d4128d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.443387 772 raft_consensus.cc:2388] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.443504 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.443598 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.443640 765 raft_consensus.cc:2463] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.443720 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.443879 768 raft_consensus.cc:3054] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.444042 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.444133 710 leader_election.cc:304] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.444214 766 raft_consensus.cc:2463] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.444399 757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ef01b41b6140d6bdad07bc79dd7dc5" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.444449 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.444490 710 leader_election.cc:304] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.444813 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.444977 760 raft_consensus.cc:2463] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.445009 710 leader_election.cc:304] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.445596 2748 raft_consensus.cc:2798] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.445686 2748 raft_consensus.cc:491] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.445756 2748 raft_consensus.cc:3054] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.444868 2886 raft_consensus.cc:2798] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.446043 2886 raft_consensus.cc:491] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.446100 2886 raft_consensus.cc:3054] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.446333 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d97ac93467a4a7a8ae44d52719f191c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.453616 1163 tablet_bootstrap.cc:492] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.453770 1163 ts_tablet_manager.cc:1397] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250112 20:51:47.454676 775 raft_consensus.cc:2463] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:47.454497 1163 raft_consensus.cc:357] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.454789 1163 raft_consensus.cc:383] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.454859 1163 raft_consensus.cc:738] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.455086 1163 consensus_queue.cc:260] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.455065 2802 raft_consensus.cc:513] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.455577 1163 ts_tablet_manager.cc:1428] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:47.455729 2802 leader_election.cc:290] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.455837 1163 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.456032 2787 raft_consensus.cc:513] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.456699 2787 leader_election.cc:290] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.457125 1186 ts_tablet_manager.cc:1918] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.457257 1186 log.cc:1198] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:47.457482 1163 tablet_bootstrap.cc:654] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.457588 1186 ts_tablet_manager.cc:1939] T 4d29316b9d2441bca97f56bdb0e56f0d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:47.458557 2750 raft_consensus.cc:2798] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.458654 2750 raft_consensus.cc:491] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.458727 2750 raft_consensus.cc:3054] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.459367 2632 consensus_queue.cc:1035] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.459682 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.459928 1041 raft_consensus.cc:3054] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.460472 710 leader_election.cc:304] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.461153 2928 raft_consensus.cc:2798] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.461259 2928 raft_consensus.cc:695] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:47.461476 2928 consensus_queue.cc:237] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.462568 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.462762 913 raft_consensus.cc:3054] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.468672 1164 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.468855 1164 ts_tablet_manager.cc:1397] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250112 20:51:47.469748 1164 raft_consensus.cc:357] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.469945 1164 raft_consensus.cc:383] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.470009 1164 raft_consensus.cc:738] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.470188 1164 consensus_queue.cc:260] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.470866 1164 ts_tablet_manager.cc:1428] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:47.471133 1164 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.471423 710 leader_election.cc:304] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.472296 2632 raft_consensus.cc:2798] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.472396 2632 raft_consensus.cc:695] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:47.472728 1164 tablet_bootstrap.cc:654] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.472602 2632 consensus_queue.cc:237] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.478685 2757 server_base.cc:1034] running on GCE node
I20250112 20:51:47.483947 710 leader_election.cc:304] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.488503 2936 raft_consensus.cc:2798] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.488663 2936 raft_consensus.cc:491] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.488829 2936 raft_consensus.cc:3054] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.489535 1164 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.489675 1164 ts_tablet_manager.cc:1397] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250112 20:51:47.490538 1164 raft_consensus.cc:357] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.490772 1164 raft_consensus.cc:383] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.490823 1164 raft_consensus.cc:738] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.490993 1164 consensus_queue.cc:260] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.491348 1178 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:47.491478 1178 ts_tablet_manager.cc:1397] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.038s
I20250112 20:51:47.492063 1178 raft_consensus.cc:357] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.492265 1178 raft_consensus.cc:383] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.492322 1178 raft_consensus.cc:738] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.492480 1163 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.492527 1178 consensus_queue.cc:260] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.492658 1163 ts_tablet_manager.cc:1397] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250112 20:51:47.493920 1163 raft_consensus.cc:357] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.494271 1163 raft_consensus.cc:383] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.494356 1163 raft_consensus.cc:738] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.494647 1163 consensus_queue.cc:260] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.495309 1163 ts_tablet_manager.cc:1428] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:47.497905 1163 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.499992 1163 tablet_bootstrap.cc:654] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.501408 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 874) took 1981 ms. Trace:
I20250112 20:51:47.501514 2447 rpcz_store.cc:276] 0112 20:51:45.519776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.521744 (+ 1968us) service_pool.cc:225] Handling call
0112 20:51:47.060030 (+1538286us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:47.060039 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.169332 (+109293us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.351095 (+181763us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.501372 (+150277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538122}
I20250112 20:51:47.501794 1164 ts_tablet_manager.cc:1428] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:47.501891 2447 tablet_replica.cc:331] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:47.502112 1164 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.502106 2447 raft_consensus.cc:2238] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.502274 2447 raft_consensus.cc:2267] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:47.505867 2949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:47.507265 2757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:47.507328 2447 ts_tablet_manager.cc:1905] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.507292 710 leader_election.cc:304] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.508764 2632 raft_consensus.cc:2798] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.508913 2632 raft_consensus.cc:491] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.509042 2632 raft_consensus.cc:3054] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.509675 1045 raft_consensus.cc:3054] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.512917 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e9f669a739d94f99a5b7e9ccccd29545 (table default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]) successfully deleted
I20250112 20:51:47.513242 1164 tablet_bootstrap.cc:654] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.514824 1163 tablet_bootstrap.cc:492] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.515077 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43ae987b5bce459c8214a0ea34b9df01" } new_table_name: "default.Table_0a39583e302f435bb0ae3f12b56e8e51" modify_external_catalogs: true
I20250112 20:51:47.519727 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet f143227e79844a7393b323222bf7eaef (table default.Table_d4e052ce75a047f98d55947e6cb75b67 [id=467b97b4e4694b2f8551e237429bd0d3]). Expected version 1 got 0
I20250112 20:51:47.521322 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8f0fb6214ee84e8a88e5292e5db7756d" } new_table_name: "default.Table_8dc3d4d22c54413aad3f7479f38380e5" modify_external_catalogs: true
I20250112 20:51:47.526959 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.527174 908 raft_consensus.cc:2463] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.527796 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.527992 1047 raft_consensus.cc:2463] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.528481 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:47.528761 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 1e584b761d234c5aa0ab875acd8e1e05 (table default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a]). Expected version 4 got 0
I20250112 20:51:47.529223 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_3591211d8ed5415b820beae2848ac1f5 [id=8133cc129a404fb8aca53d1fdb1c7fef] alter complete (version 3)
W20250112 20:51:47.530495 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:47943 (request call id 65) took 774 ms (client timeout 1000 ms). Trace:
W20250112 20:51:47.530592 447 rpcz_store.cc:269] 0112 20:51:46.756491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.756918 (+ 427us) service_pool.cc:225] Handling call
0112 20:51:47.222795 (+465877us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:47.222804 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:47.530428 (+307624us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:47.061977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:47.062076 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:47.062083 (+ 7us) write_op.cc:435] Acquired schema lock
0112 20:51:47.062085 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:47.062187 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:47.062201 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:47.062206 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:47.062268 (+ 62us) tablet.cc:698] Row locks acquired
0112 20:51:47.062271 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:47.062306 (+ 35us) write_op.cc:270] Start()
0112 20:51:47.062337 (+ 31us) write_op.cc:276] Timestamp: P: 1736715107062253 usec, L: 0
0112 20:51:47.062340 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:47.062512 (+ 172us) log.cc:844] Serialized 356 byte log entry
0112 20:51:47.066717 (+ 4205us) op_driver.cc:464] REPLICATION: finished
0112 20:51:47.066854 (+ 137us) write_op.cc:301] APPLY: starting
0112 20:51:47.066869 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:47.066910 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:47.066935 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:47.066984 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:47.067015 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:47.067022 (+ 7us) write_op.cc:312] APPLY: finished
0112 20:51:47.067088 (+ 66us) log.cc:844] Serialized 23 byte log entry
0112 20:51:47.067130 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:47.067144 (+ 14us) write_op.cc:454] Released schema lock
0112 20:51:47.067150 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":11,"spinlock_wait_cycles":768,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":341,"num_ops":1,"prepare.queue_time_us":131203,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":662,"raft.queue_time_us":428,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":256,"replication_time_us":4349}]]}
I20250112 20:51:47.531065 847 leader_election.cc:304] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.531435 2802 raft_consensus.cc:2798] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.531535 2802 raft_consensus.cc:695] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:47.531741 2802 consensus_queue.cc:237] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:47.533201 847 leader_election.cc:343] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.533301 847 leader_election.cc:304] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.533588 2521 raft_consensus.cc:2743] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:47.533963 847 leader_election.cc:343] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.534049 847 leader_election.cc:304] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.534359 2521 raft_consensus.cc:2743] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.535315 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_16215b3f2b9d4bba913676a2f2590409"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.535820 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16215b3f2b9d4bba913676a2f2590409 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.536039 1164 tablet_bootstrap.cc:492] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.536168 1164 ts_tablet_manager.cc:1397] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20250112 20:51:47.539080 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.539288 908 raft_consensus.cc:2388] T 0093059b34fb40609289f578978c8750 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:47.539969 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f4d062a92b0d473e8d60d7224cbc4f9c" } new_table_name: "default.Table_db1ef1dc5c51459f81699155e442a956" modify_external_catalogs: true
W20250112 20:51:47.540334 847 leader_election.cc:343] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.540436 847 leader_election.cc:304] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.540642 2521 raft_consensus.cc:2743] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.541034 2521 consensus_queue.cc:1035] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.541584 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0093059b34fb40609289f578978c8750" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.543653 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 23e81fd99c60408c911dee56146f4735 (DEFAULT_TABLE table=default.Table_2e76748510074c8f882cd75a58be6dbc [id=1658ffb2fa504c768979ada418098d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.543848 1014 tablet_service.cc:1467] Processing CreateTablet for tablet c18bbd6e86f54cbb9749ac36c1f726da (DEFAULT_TABLE table=default.Table_28ec28b7ea3048b6adc23abf2325689a [id=03c00b2d93764d5e9dd4a8ce9c2bbc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.544147 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 (DEFAULT_TABLE table=default.Table_73ac32d005e74ff68df62ba8e7cc94ff [id=f51ee475b9bb47068c292ef9f57ecbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.544636 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 630427adafd247578a31c900fa242cdd (DEFAULT_TABLE table=default.Table_53972fb4ced04252bb122116083a5c9b [id=f7e31439aed748ebbf1556c8ef59addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.544816 983 leader_election.cc:304] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.545336 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 697048b92a96463a8a3d7dbee489a66d (DEFAULT_TABLE table=default.Table_0515c2f2d9b346fe99c0cea8663c500a [id=ad9a7cc1bb644e8bb0052309d696f059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.545591 983 leader_election.cc:304] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.545917 2926 raft_consensus.cc:2743] T f143227e79844a7393b323222bf7eaef P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.545970 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 01bad472601a42268bacb63116352f75 (DEFAULT_TABLE table=default.Table_d308f6e26ddd443c9a3b7c3f69a7bfc1 [id=e04475b960b147838d5d5a65c0ca81b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.546099 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2558fdfd08c3453388479a54cf788f01 (DEFAULT_TABLE table=default.Table_09a9e02cfb4c4688afd3b2a20584f610 [id=9f840a2974d549f38f8e91e069abcbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.546432 983 leader_election.cc:304] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.546783 2926 raft_consensus.cc:3054] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.546905 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3343e1d7f1d04a498bd0ee00311b6a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.546788 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.547237 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.547333 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6b4882a3d744bda5677aa9e01a53f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6921185b545244978f8fdeafd1fb6c5e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.547621 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d0018e1eabf24241a358a7e00b6b5907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:32 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.547627 983 leader_election.cc:304] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.548032 2933 raft_consensus.cc:3054] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.548555 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8bea75e61b954ec8b0838e2d6fa87628"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.549129 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bea75e61b954ec8b0838e2d6fa87628 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.551106 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 890) took 2637 ms. Trace:
I20250112 20:51:47.551227 1186 rpcz_store.cc:276] 0112 20:51:44.913710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.914312 (+ 602us) service_pool.cc:225] Handling call
0112 20:51:47.206128 (+2291816us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:47.206139 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.365487 (+159348us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.457003 (+ 91516us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.551069 (+ 94066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2291717}
I20250112 20:51:47.551571 1186 tablet_replica.cc:331] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:47.552376 1163 ts_tablet_manager.cc:1397] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.003s
I20250112 20:51:47.552642 911 raft_consensus.cc:2463] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.547685 1164 raft_consensus.cc:357] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.555034 1164 raft_consensus.cc:383] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.555115 1164 raft_consensus.cc:738] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.555220 913 raft_consensus.cc:2463] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.555544 2927 raft_consensus.cc:513] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.555708 1041 raft_consensus.cc:2463] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:47.556156 772 raft_consensus.cc:2388] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:47.556216 2927 leader_election.cc:290] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.557058 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18bbd6e86f54cbb9749ac36c1f726da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.557168 1186 raft_consensus.cc:2238] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.557436 1186 raft_consensus.cc:2267] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.557603 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01bad472601a42268bacb63116352f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.557956 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2558fdfd08c3453388479a54cf788f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.557968 1186 ts_tablet_manager.cc:1905] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.558341 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630427adafd247578a31c900fa242cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.558571 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146994989d6d4ce8a12e1b8677d7feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.560956 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e81fd99c60408c911dee56146f4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.561617 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697048b92a96463a8a3d7dbee489a66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.564723 1164 consensus_queue.cc:260] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.565321 1163 raft_consensus.cc:357] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.565497 1163 raft_consensus.cc:383] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.565565 1163 raft_consensus.cc:738] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.565721 1163 consensus_queue.cc:260] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.566272 983 leader_election.cc:304] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.567062 2963 raft_consensus.cc:491] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.567253 2963 raft_consensus.cc:513] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.568256 2963 leader_election.cc:290] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.568559 2963 raft_consensus.cc:2798] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.568720 2963 raft_consensus.cc:695] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.569077 2963 consensus_queue.cc:237] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.574784 2757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:47.576202 2757 hybrid_clock.cc:648] HybridClock initialized: now 1736715107576102 us; error 77 us; skew 500 ppm
I20250112 20:51:47.576853 2757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:47.577836 983 leader_election.cc:304] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.578550 1164 ts_tablet_manager.cc:1428] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
W20250112 20:51:47.578910 847 leader_election.cc:343] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.579018 847 leader_election.cc:304] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.579386 847 leader_election.cc:304] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.579588 2108 raft_consensus.cc:2743] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.579723 2108 raft_consensus.cc:2798] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.579782 2108 raft_consensus.cc:491] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.579831 2108 raft_consensus.cc:3054] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:47.583041 983 leader_election.cc:343] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:47.588564 1164 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:47.579720 847 leader_election.cc:343] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.588755 847 leader_election.cc:304] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.589875 847 leader_election.cc:304] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:47.590267 1164 tablet_bootstrap.cc:654] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.591342 2927 raft_consensus.cc:491] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.591457 2927 raft_consensus.cc:513] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.591956 2927 leader_election.cc:290] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.593762 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.594587 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.595374 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.595724 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.597184 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.597332 908 raft_consensus.cc:2463] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.598461 2975 raft_consensus.cc:2743] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.598927 877 tablet_service.cc:1467] Processing CreateTablet for tablet 23e81fd99c60408c911dee56146f4735 (DEFAULT_TABLE table=default.Table_2e76748510074c8f882cd75a58be6dbc [id=1658ffb2fa504c768979ada418098d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.598982 888 tablet_service.cc:1467] Processing CreateTablet for tablet c18bbd6e86f54cbb9749ac36c1f726da (DEFAULT_TABLE table=default.Table_28ec28b7ea3048b6adc23abf2325689a [id=03c00b2d93764d5e9dd4a8ce9c2bbc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.599325 710 leader_election.cc:304] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.599400 875 tablet_service.cc:1467] Processing CreateTablet for tablet 630427adafd247578a31c900fa242cdd (DEFAULT_TABLE table=default.Table_53972fb4ced04252bb122116083a5c9b [id=f7e31439aed748ebbf1556c8ef59addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.599400 874 tablet_service.cc:1467] Processing CreateTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 (DEFAULT_TABLE table=default.Table_73ac32d005e74ff68df62ba8e7cc94ff [id=f51ee475b9bb47068c292ef9f57ecbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.599671 2975 raft_consensus.cc:2798] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.599757 2975 raft_consensus.cc:695] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:47.585000 2966 raft_consensus.cc:2798] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.599897 983 leader_election.cc:304] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.599961 2966 raft_consensus.cc:695] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.599996 2975 consensus_queue.cc:237] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.600225 889 tablet_service.cc:1467] Processing CreateTablet for tablet 697048b92a96463a8a3d7dbee489a66d (DEFAULT_TABLE table=default.Table_0515c2f2d9b346fe99c0cea8663c500a [id=ad9a7cc1bb644e8bb0052309d696f059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.585536 1163 ts_tablet_manager.cc:1428] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.033s user 0.003s sys 0.002s
I20250112 20:51:47.600301 2968 raft_consensus.cc:2743] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.600548 1163 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.600841 983 leader_election.cc:304] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.600925 878 tablet_service.cc:1467] Processing CreateTablet for tablet 01bad472601a42268bacb63116352f75 (DEFAULT_TABLE table=default.Table_d308f6e26ddd443c9a3b7c3f69a7bfc1 [id=e04475b960b147838d5d5a65c0ca81b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.601158 2968 raft_consensus.cc:2743] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.601714 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2558fdfd08c3453388479a54cf788f01 (DEFAULT_TABLE table=default.Table_09a9e02cfb4c4688afd3b2a20584f610 [id=9f840a2974d549f38f8e91e069abcbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.601708 2968 consensus_queue.cc:1035] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.602020 1163 tablet_bootstrap.cc:654] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.602447 983 leader_election.cc:304] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.602485 884 tablet_service.cc:1467] Processing CreateTablet for tablet b2b14d1374c342cf84374122f8a865dc (DEFAULT_TABLE table=default.Table_fec3ae6413364d4bb016e523ce9736aa [id=78a0edb144f64f12950ee5daf798cb78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.600319 2966 consensus_queue.cc:237] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.602800 2968 raft_consensus.cc:2743] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.603046 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 4d29316b9d2441bca97f56bdb0e56f0d (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
I20250112 20:51:47.605168 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.605418 908 raft_consensus.cc:2388] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:47.614216 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b8cc99742f4f9ba68eb1aa17cab248" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.615671 2927 raft_consensus.cc:2798] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.615675 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.615772 2927 raft_consensus.cc:491] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.615840 2927 raft_consensus.cc:3054] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.615828 1041 raft_consensus.cc:2463] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.620486 2757 webserver.cc:458] Webserver started at http://127.0.104.61:39277/ using document root <none> and password file <none>
I20250112 20:51:47.622362 2757 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:47.622514 2757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:47.629485 2955 raft_consensus.cc:491] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.629832 2955 raft_consensus.cc:513] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.630438 2955 leader_election.cc:290] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:47.632277 710 consensus_peers.cc:487] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01367a11dea04b33a9255508501b9060. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:47.634964 1163 tablet_bootstrap.cc:492] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.635118 1163 ts_tablet_manager.cc:1397] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
W20250112 20:51:47.635591 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2363610c772b4a92bce5d4428f5f5c8e consensus state has no leader
I20250112 20:51:47.635654 1178 ts_tablet_manager.cc:1428] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.144s user 0.001s sys 0.001s
I20250112 20:51:47.635890 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 438e2c14353c4a8cb7242d75bb3ba673 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:47.635809 1163 raft_consensus.cc:357] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.636008 1163 raft_consensus.cc:383] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.636062 1163 raft_consensus.cc:738] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.636229 1163 consensus_queue.cc:260] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.636404 1178 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:47.636801 1163 ts_tablet_manager.cc:1428] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:47.638532 1164 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.638576 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cdcbd0285dd04edebfa92654c16e5d39" } modify_external_catalogs: true
I20250112 20:51:47.638659 1164 ts_tablet_manager.cc:1397] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20250112 20:51:47.638756 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cdcbd0285dd04edebfa92654c16e5d39" } modify_external_catalogs: true
I20250112 20:51:47.639288 1164 raft_consensus.cc:357] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.639475 1164 raft_consensus.cc:383] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.639518 1164 raft_consensus.cc:738] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.637084 1163 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.641449 710 leader_election.cc:304] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:47.642330 1011 tablet_service.cc:1467] Processing CreateTablet for tablet b2b14d1374c342cf84374122f8a865dc (DEFAULT_TABLE table=default.Table_fec3ae6413364d4bb016e523ce9736aa [id=78a0edb144f64f12950ee5daf798cb78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.642450 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c54516970d424f7cb28c72d8fe52b824 (DEFAULT_TABLE table=default.Table_2ac8449ec8b94e0494cceb60303f0481 [id=3ca64f2a762445789e9a422e02f67cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.643100 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ffe14c8f0406451bae99014c98c6cae1 (DEFAULT_TABLE table=default.Table_d9a46ff1902a48529ee58d8c36e0d9e0 [id=2c0c563d63d147b2b7e791413c988823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.643199 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc545822e98460e815df4e6cad455b4 (DEFAULT_TABLE table=default.Table_96d7128d31fe403abd7f96408718800c [id=2b3b843a4c534104acefa550d9af6545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.647545 2886 raft_consensus.cc:513] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.648332 2886 leader_election.cc:290] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.653419 2750 raft_consensus.cc:513] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.654176 2750 leader_election.cc:290] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.655498 1163 tablet_bootstrap.cc:654] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.658800 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e81fd99c60408c911dee56146f4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.659381 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18bbd6e86f54cbb9749ac36c1f726da. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:47.659760 710 leader_election.cc:343] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 20:51:47.659806 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630427adafd247578a31c900fa242cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.660228 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146994989d6d4ce8a12e1b8677d7feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.660410 710 leader_election.cc:304] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.660600 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697048b92a96463a8a3d7dbee489a66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.660665 1164 consensus_queue.cc:260] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.660979 2750 raft_consensus.cc:2798] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.660998 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2558fdfd08c3453388479a54cf788f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.661046 2750 raft_consensus.cc:491] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.661104 2750 raft_consensus.cc:3054] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.661379 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b14d1374c342cf84374122f8a865dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:47.662108 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5797c2fd8e21490ca00ef171965c31ad consensus state has no leader
I20250112 20:51:47.662918 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0bb9c2e6f37f4184a106aed05bae7924
I20250112 20:51:47.663352 454 catalog_manager.cc:6596] ReplaceTablet: tablet 0bb9c2e6f37f4184a106aed05bae7924 deleted and replaced by tablet c17e8808d62d4b7da88bffe7d7569568
I20250112 20:51:47.663928 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01bad472601a42268bacb63116352f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.664848 1164 ts_tablet_manager.cc:1428] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250112 20:51:47.665207 1164 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.666601 1164 tablet_bootstrap.cc:654] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.669128 775 raft_consensus.cc:2388] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:47.669095 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.669284 911 raft_consensus.cc:2463] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.669840 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 9705feb15f384b5185935eacc521d0e0
I20250112 20:51:47.669948 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.670079 911 raft_consensus.cc:3054] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.670328 446 catalog_manager.cc:6596] ReplaceTablet: tablet 9705feb15f384b5185935eacc521d0e0 deleted and replaced by tablet 1596ecc13e4a4de58315a28d04697a1b
I20250112 20:51:47.670732 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.670935 913 raft_consensus.cc:2388] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:47.668731 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7d9572179a6f45759214c6c39f22b2f0
I20250112 20:51:47.674315 452 master_service.cc:946] ReplaceTablet: received request to replace tablet c6bdfecd5d9142d5b008c613864c8217 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:47.685582 710 leader_election.cc:304] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:47.686095 2971 raft_consensus.cc:491] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.686220 2971 raft_consensus.cc:513] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.686913 2971 leader_election.cc:290] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.687608 2955 raft_consensus.cc:2743] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.690647 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 6b440da020d947dda2553d65dd6a4bdf from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:47.691076 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8
W20250112 20:51:47.692809 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e137112722545bbae406cf54347aa8c consensus state has no leader
I20250112 20:51:47.693048 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e72a6c8e18934b789a048d5aaded121b" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.693583 2563 raft_consensus.cc:513] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.694252 2563 leader_election.cc:290] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.694794 2108 raft_consensus.cc:513] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.695104 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1344f1b683b941899cf95663d76ffd18" } modify_external_catalogs: true
I20250112 20:51:47.695262 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1344f1b683b941899cf95663d76ffd18" } modify_external_catalogs: true
I20250112 20:51:47.695410 2108 leader_election.cc:290] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.696655 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_78ebfe3d8c41485999992166d9bf2149"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.697247 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78ebfe3d8c41485999992166d9bf2149 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.697331 1164 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.697448 1164 ts_tablet_manager.cc:1397] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250112 20:51:47.697801 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.697964 908 raft_consensus.cc:3054] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.698098 710 leader_election.cc:304] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.698591 2955 raft_consensus.cc:2798] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.698530 2563 consensus_queue.cc:1035] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.698648 2955 raft_consensus.cc:491] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.698705 2955 raft_consensus.cc:3054] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.699537 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c39266c8bbc54b86921b055275242fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.699851 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0d68049237d04746aed93d5313f67bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.700099 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c39266c8bbc54b86921b055275242fc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:47.700218 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d68049237d04746aed93d5313f67bc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.701395 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_73ac32d005e74ff68df62ba8e7cc94ff" } modify_external_catalogs: true
I20250112 20:51:47.701545 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_73ac32d005e74ff68df62ba8e7cc94ff" } modify_external_catalogs: true
I20250112 20:51:47.701799 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.701728 1164 raft_consensus.cc:357] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.701964 1164 raft_consensus.cc:383] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.702024 1164 raft_consensus.cc:738] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:47.702284 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.703897 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:47.704319 1163 tablet_bootstrap.cc:492] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.704504 983 leader_election.cc:304] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.704919 2563 raft_consensus.cc:3054] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.704983 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_688c94cdb32f468587b57b9c7e9a895c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.704468 1163 ts_tablet_manager.cc:1397] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.009s
W20250112 20:51:47.705456 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_688c94cdb32f468587b57b9c7e9a895c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.705803 1163 raft_consensus.cc:357] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.705961 1163 raft_consensus.cc:383] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.706027 1163 raft_consensus.cc:738] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.706211 1163 consensus_queue.cc:260] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.708197 1163 ts_tablet_manager.cc:1428] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:47.708508 1163 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.710242 1163 tablet_bootstrap.cc:654] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.712280 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 438e2c14353c4a8cb7242d75bb3ba673
I20250112 20:51:47.712684 449 catalog_manager.cc:6596] ReplaceTablet: tablet 438e2c14353c4a8cb7242d75bb3ba673 deleted and replaced by tablet 08b293868ae546c88f393b17e51147d0
I20250112 20:51:47.714479 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b062c1d67d4443eabe8ac1d8a832c243
I20250112 20:51:47.717471 1164 consensus_queue.cc:260] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.719339 1164 ts_tablet_manager.cc:1428] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250112 20:51:47.719621 1164 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.722743 1164 tablet_bootstrap.cc:654] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.727607 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b14d1374c342cf84374122f8a865dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.727844 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54516970d424f7cb28c72d8fe52b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.728291 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc545822e98460e815df4e6cad455b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.728446 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe14c8f0406451bae99014c98c6cae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.729508 1163 tablet_bootstrap.cc:492] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.729616 1163 ts_tablet_manager.cc:1397] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250112 20:51:47.730197 1163 raft_consensus.cc:357] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.730370 1163 raft_consensus.cc:383] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.730413 1163 raft_consensus.cc:738] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.730592 1163 consensus_queue.cc:260] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.731490 2447 ts_tablet_manager.cc:1918] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.731596 2447 log.cc:1198] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/921ae136d5e2438e950b96f156f23da3
I20250112 20:51:47.731935 2447 ts_tablet_manager.cc:1939] T 921ae136d5e2438e950b96f156f23da3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:47.732163 2757 fs_manager.cc:714] Time spent opening directory manager: real 0.101s user 0.005s sys 0.000s
I20250112 20:51:47.736899 1178 tablet_bootstrap.cc:654] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.741148 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4a7a33e2dd7b42ada35718df1413bdd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.741708 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a7a33e2dd7b42ada35718df1413bdd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.742825 2787 raft_consensus.cc:2743] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.743096 1185 ts_tablet_manager.cc:1918] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.743216 1185 log.cc:1198] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/89f9b46da06a476ab9f2a93ca74ff736
I20250112 20:51:47.743543 1185 ts_tablet_manager.cc:1939] T 89f9b46da06a476ab9f2a93ca74ff736 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:47.743752 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ae66395240e9403a94131921dfe99d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:47.743805 2750 raft_consensus.cc:513] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.744009 2108 raft_consensus.cc:491] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.744129 2108 raft_consensus.cc:513] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:47.744222 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae66395240e9403a94131921dfe99d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.744467 2750 leader_election.cc:290] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.744644 2108 leader_election.cc:290] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.745115 1163 ts_tablet_manager.cc:1428] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250112 20:51:47.745366 1163 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.746433 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 226ea7f38f1445c598676715a3036227
I20250112 20:51:47.747134 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d95d780f503949e3b24983e2c1202f02" } new_table_name: "default.Table_162179129dc94078b87f0bf9820e6253" modify_external_catalogs: true
I20250112 20:51:47.748071 2954 raft_consensus.cc:491] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.748188 2954 raft_consensus.cc:513] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.748741 1163 tablet_bootstrap.cc:654] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.749462 908 raft_consensus.cc:2463] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.750386 1186 ts_tablet_manager.cc:1918] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:47.750509 1186 log.cc:1198] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/49b8cc99742f4f9ba68eb1aa17cab248
I20250112 20:51:47.750783 1186 ts_tablet_manager.cc:1939] T 49b8cc99742f4f9ba68eb1aa17cab248 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:47.751278 2954 leader_election.cc:290] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.754812 1164 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.754945 1164 ts_tablet_manager.cc:1397] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250112 20:51:47.755565 1164 raft_consensus.cc:357] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.755733 1164 raft_consensus.cc:383] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.755777 1164 raft_consensus.cc:738] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.760282 1164 consensus_queue.cc:260] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.760562 983 leader_election.cc:304] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.761052 2963 raft_consensus.cc:2798] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.761190 2963 raft_consensus.cc:695] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.761459 1164 ts_tablet_manager.cc:1428] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:47.761436 2963 consensus_queue.cc:237] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.761809 1164 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.763278 1163 tablet_bootstrap.cc:492] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.763389 1163 ts_tablet_manager.cc:1397] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20250112 20:51:47.764055 1163 raft_consensus.cc:357] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.764176 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6b440da020d947dda2553d65dd6a4bdf
I20250112 20:51:47.764257 1163 raft_consensus.cc:383] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.764333 1163 raft_consensus.cc:738] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.765014 1163 consensus_queue.cc:260] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.765384 447 catalog_manager.cc:6596] ReplaceTablet: tablet 6b440da020d947dda2553d65dd6a4bdf deleted and replaced by tablet 2e163a5b99a44d34bf59ada02c2bba99
I20250112 20:51:47.765295 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.765507 908 raft_consensus.cc:3054] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.766100 1163 ts_tablet_manager.cc:1428] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:47.766263 2963 consensus_queue.cc:1035] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.766371 1163 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.766309 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.766462 913 raft_consensus.cc:2463] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.766973 710 leader_election.cc:304] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:47.767227 2954 raft_consensus.cc:2798] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.767305 2954 raft_consensus.cc:491] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.767364 2954 raft_consensus.cc:3054] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.770778 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f669f2f5bed64a6280189cdee98bc64d (DEFAULT_TABLE table=default.Table_d95d780f503949e3b24983e2c1202f02 [id=54217c15618341679237c44c98af621d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.771093 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.771194 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f669f2f5bed64a6280189cdee98bc64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.771339 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 226ea7f38f1445c598676715a3036227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.771122 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 with delete_type TABLET_DATA_DELETED (replaced by tablet 08b293868ae546c88f393b17e51147d0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.771699 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b440da020d947dda2553d65dd6a4bdf with delete_type TABLET_DATA_DELETED (replaced by tablet 2e163a5b99a44d34bf59ada02c2bba99) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.771978 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd47d1c1a4b42bdbff97f619dc95134" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.771970 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.772171 1041 raft_consensus.cc:3054] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.772244 1045 raft_consensus.cc:2463] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.772722 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669e1cfd407e4d87a63e6832d9c6fa1f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.772905 1049 raft_consensus.cc:3054] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.773102 1025 tablet_service.cc:1467] Processing CreateTablet for tablet bdcd45226ab44353b5574bfd4dae7f9a (DEFAULT_TABLE table=default.Table_132455edcc094422bdf769a1c836cf16 [id=db8fd22edd3b4d4fa88e1f6e1273d666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.773422 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcd45226ab44353b5574bfd4dae7f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.773702 1163 tablet_bootstrap.cc:654] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.776249 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.776388 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e8808d62d4b7da88bffe7d7569568) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.776464 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56544c3b0cf140089baa678765a2eb79" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.776554 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 9705feb15f384b5185935eacc521d0e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1596ecc13e4a4de58315a28d04697a1b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.776388 1042 raft_consensus.cc:2463] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:47.776726 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.776686 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.779629 1164 tablet_bootstrap.cc:654] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.787488 1163 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.787518 2963 raft_consensus.cc:491] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.787626 1163 ts_tablet_manager.cc:1397] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250112 20:51:47.787784 2963 raft_consensus.cc:513] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.788479 1163 raft_consensus.cc:357] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.788726 1163 raft_consensus.cc:383] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.789654 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.789901 913 raft_consensus.cc:2388] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.790000 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.790175 761 raft_consensus.cc:2463] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.788997 2963 leader_election.cc:290] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.790875 983 leader_election.cc:304] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:47.791517 1163 raft_consensus.cc:738] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.791539 2787 raft_consensus.cc:491] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.791646 2787 raft_consensus.cc:513] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.791868 1164 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:47.791910 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.791986 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 577dc789f0cb48f49168124b37a6fba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.792068 756 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.792167 2787 leader_election.cc:290] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.792215 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.792301 2787 raft_consensus.cc:2798] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.792363 2787 raft_consensus.cc:491] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.792344 738 tablet_service.cc:1467] Processing CreateTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 (DEFAULT_TABLE table=default.Table_73ac32d005e74ff68df62ba8e7cc94ff [id=f51ee475b9bb47068c292ef9f57ecbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.792426 2787 raft_consensus.cc:3054] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.792449 754 tablet_service.cc:1467] Processing CreateTablet for tablet c18bbd6e86f54cbb9749ac36c1f726da (DEFAULT_TABLE table=default.Table_28ec28b7ea3048b6adc23abf2325689a [id=03c00b2d93764d5e9dd4a8ce9c2bbc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.792699 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146994989d6d4ce8a12e1b8677d7feb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.792881 756 tablet_service.cc:1467] Processing CreateTablet for tablet 630427adafd247578a31c900fa242cdd (DEFAULT_TABLE table=default.Table_53972fb4ced04252bb122116083a5c9b [id=f7e31439aed748ebbf1556c8ef59addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.792994 745 tablet_service.cc:1467] Processing CreateTablet for tablet 23e81fd99c60408c911dee56146f4735 (DEFAULT_TABLE table=default.Table_2e76748510074c8f882cd75a58be6dbc [id=1658ffb2fa504c768979ada418098d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.793182 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 630427adafd247578a31c900fa242cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.793375 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e81fd99c60408c911dee56146f4735. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.794770 737 tablet_service.cc:1467] Processing CreateTablet for tablet 697048b92a96463a8a3d7dbee489a66d (DEFAULT_TABLE table=default.Table_0515c2f2d9b346fe99c0cea8663c500a [id=ad9a7cc1bb644e8bb0052309d696f059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.795055 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697048b92a96463a8a3d7dbee489a66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.796034 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c18bbd6e86f54cbb9749ac36c1f726da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.799125 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.799465 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229e4ef771b4032a4bfae3190a851e3" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:47.791776 1163 consensus_queue.cc:260] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.799875 983 leader_election.cc:304] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.800226 1163 ts_tablet_manager.cc:1428] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250112 20:51:47.800318 2968 raft_consensus.cc:2743] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.800482 1163 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.804040 1178 tablet_bootstrap.cc:492] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:47.804522 710 leader_election.cc:304] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:47.805403 747 tablet_service.cc:1467] Processing CreateTablet for tablet 01bad472601a42268bacb63116352f75 (DEFAULT_TABLE table=default.Table_d308f6e26ddd443c9a3b7c3f69a7bfc1 [id=e04475b960b147838d5d5a65c0ca81b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.805748 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01bad472601a42268bacb63116352f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.806762 1163 tablet_bootstrap.cc:654] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.809094 739 tablet_service.cc:1467] Processing CreateTablet for tablet 2558fdfd08c3453388479a54cf788f01 (DEFAULT_TABLE table=default.Table_09a9e02cfb4c4688afd3b2a20584f610 [id=9f840a2974d549f38f8e91e069abcbcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.809448 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2558fdfd08c3453388479a54cf788f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.809473 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_137d3f9986ea49519c9585b3761b586c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:47.810041 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_137d3f9986ea49519c9585b3761b586c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.812973 752 tablet_service.cc:1467] Processing CreateTablet for tablet b2b14d1374c342cf84374122f8a865dc (DEFAULT_TABLE table=default.Table_fec3ae6413364d4bb016e523ce9736aa [id=78a0edb144f64f12950ee5daf798cb78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.813372 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b14d1374c342cf84374122f8a865dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.813367 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_743de41614e248bd9cffca489bc49570" } new_table_name: "default.Table_a1de4aac1b4f453dbbd50718c2d98dcf" modify_external_catalogs: true
I20250112 20:51:47.814486 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:47.814674 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:47.814848 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:47.815786 451 catalog_manager.cc:5526] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:47.816229 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
I20250112 20:51:47.816453 451 catalog_manager.cc:5526] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:47.816905 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:47.817828 2950 raft_consensus.cc:2743] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.818735 1164 ts_tablet_manager.cc:1397] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.000s
I20250112 20:51:47.819468 1164 raft_consensus.cc:357] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.819685 1164 raft_consensus.cc:383] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.819756 1164 raft_consensus.cc:738] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.820014 1164 consensus_queue.cc:260] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.820523 1164 ts_tablet_manager.cc:1428] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:47.820813 1164 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:47.820902 1163 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.821015 1163 ts_tablet_manager.cc:1397] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250112 20:51:47.822141 1163 raft_consensus.cc:357] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.822319 1163 raft_consensus.cc:383] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.822365 1163 raft_consensus.cc:738] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.822541 1163 consensus_queue.cc:260] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.822991 1163 ts_tablet_manager.cc:1428] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:47.823246 1163 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.824641 1163 tablet_bootstrap.cc:654] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.824620 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 146994989d6d4ce8a12e1b8677d7feb7
I20250112 20:51:47.830598 1164 tablet_bootstrap.cc:654] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.832132 3008 tablet.cc:1721] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:47.833482 2964 raft_consensus.cc:491] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.833977 2964 raft_consensus.cc:513] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.838128 768 raft_consensus.cc:2463] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.838270 2957 raft_consensus.cc:2743] T 0093059b34fb40609289f578978c8750 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.844472 2059 raft_consensus.cc:513] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.844810 772 raft_consensus.cc:2388] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:47.845131 2059 leader_election.cc:290] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.848764 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2ee684deb6634f31981eab910abbc0fb" } new_table_name: "default.Table_01a29636f6b34c918eb32d012e74e420" modify_external_catalogs: true
I20250112 20:51:47.852183 2936 raft_consensus.cc:513] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.852316 2632 raft_consensus.cc:513] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.852764 2936 leader_election.cc:290] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.853021 2748 raft_consensus.cc:513] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.854930 2748 leader_election.cc:290] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.855053 2927 raft_consensus.cc:513] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.853191 744 tablet_service.cc:1467] Processing CreateTablet for tablet ffe14c8f0406451bae99014c98c6cae1 (DEFAULT_TABLE table=default.Table_d9a46ff1902a48529ee58d8c36e0d9e0 [id=2c0c563d63d147b2b7e791413c988823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.855517 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe14c8f0406451bae99014c98c6cae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.855564 2927 leader_election.cc:290] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.857632 2955 raft_consensus.cc:513] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.858081 2955 leader_election.cc:290] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.859016 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 904) took 2457 ms. Trace:
I20250112 20:51:47.859086 1185 rpcz_store.cc:276] 0112 20:51:45.401421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.403002 (+ 1581us) service_pool.cc:225] Handling call
0112 20:51:47.426950 (+2023948us) ts_tablet_manager.cc:1145] Deleting tablet 89f9b46da06a476ab9f2a93ca74ff736
0112 20:51:47.426961 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.653024 (+226063us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.742942 (+ 89918us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.858999 (+116057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2023785}
I20250112 20:51:47.859421 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 905) took 2435 ms. Trace:
I20250112 20:51:47.859477 1185 rpcz_store.cc:276] 0112 20:51:45.424342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.426504 (+ 2162us) service_pool.cc:225] Handling call
0112 20:51:47.859299 (+2432795us) ts_tablet_manager.cc:1145] Deleting tablet 015f54bdae6b4333963f9e085c5189e8
0112 20:51:47.859307 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.859409 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432636}
I20250112 20:51:47.859756 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 906) took 2427 ms. Trace:
I20250112 20:51:47.859817 1185 rpcz_store.cc:276] 0112 20:51:45.432604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.437223 (+ 4619us) service_pool.cc:225] Handling call
0112 20:51:47.859679 (+2422456us) ts_tablet_manager.cc:1145] Deleting tablet 0d072ebae78e40c48e158eb65775476c
0112 20:51:47.859686 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.859745 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422292}
I20250112 20:51:47.860093 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 907) took 2419 ms. Trace:
I20250112 20:51:47.860152 1185 rpcz_store.cc:276] 0112 20:51:45.440995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.442668 (+ 1673us) service_pool.cc:225] Handling call
0112 20:51:47.860001 (+2417333us) ts_tablet_manager.cc:1145] Deleting tablet 3f6ad2ac50c745448031bdf94e0eea59
0112 20:51:47.860008 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.860079 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417185}
I20250112 20:51:47.860400 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 908) took 2408 ms. Trace:
I20250112 20:51:47.860448 1185 rpcz_store.cc:276] 0112 20:51:45.451489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.451545 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:47.860329 (+2408784us) ts_tablet_manager.cc:1145] Deleting tablet 69012f97aac1441095aa3bb1548de9cd
0112 20:51:47.860336 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.860390 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408628}
I20250112 20:51:47.860631 1185 tablet_replica.cc:331] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:47.860603 751 tablet_service.cc:1467] Processing CreateTablet for tablet bdcd45226ab44353b5574bfd4dae7f9a (DEFAULT_TABLE table=default.Table_132455edcc094422bdf769a1c836cf16 [id=db8fd22edd3b4d4fa88e1f6e1273d666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.861008 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcd45226ab44353b5574bfd4dae7f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.853255 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc545822e98460e815df4e6cad455b4 (DEFAULT_TABLE table=default.Table_96d7128d31fe403abd7f96408718800c [id=2b3b843a4c534104acefa550d9af6545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.861313 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc545822e98460e815df4e6cad455b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.853379 750 tablet_service.cc:1467] Processing CreateTablet for tablet f669f2f5bed64a6280189cdee98bc64d (DEFAULT_TABLE table=default.Table_d95d780f503949e3b24983e2c1202f02 [id=54217c15618341679237c44c98af621d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.862330 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f669f2f5bed64a6280189cdee98bc64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.853381 2632 leader_election.cc:290] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.865928 741 tablet_service.cc:1467] Processing CreateTablet for tablet c07855f83f484e62b400775f71925857 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.866199 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07855f83f484e62b400775f71925857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.867939 1185 raft_consensus.cc:2238] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.868084 1185 raft_consensus.cc:2267] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.868682 1185 ts_tablet_manager.cc:1905] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:47.869272 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f669f2f5bed64a6280189cdee98bc64d consensus state has no leader
I20250112 20:51:47.869427 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e8808d62d4b7da88bffe7d7569568) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.869599 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9705feb15f384b5185935eacc521d0e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1596ecc13e4a4de58315a28d04697a1b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.870641 775 raft_consensus.cc:2388] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:47.840734 2923 raft_consensus.cc:513] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.854187 753 tablet_service.cc:1467] Processing CreateTablet for tablet c54516970d424f7cb28c72d8fe52b824 (DEFAULT_TABLE table=default.Table_2ac8449ec8b94e0494cceb60303f0481 [id=3ca64f2a762445789e9a422e02f67cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.871910 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54516970d424f7cb28c72d8fe52b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.873312 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.873528 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 with delete_type TABLET_DATA_DELETED (replaced by tablet 08b293868ae546c88f393b17e51147d0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.884027 2954 raft_consensus.cc:513] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.894665 2954 leader_election.cc:290] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.895267 2923 leader_election.cc:290] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.901333 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.903873 2923 raft_consensus.cc:491] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.916864 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:47.917872 1178 ts_tablet_manager.cc:1397] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.282s user 0.002s sys 0.002s
I20250112 20:51:47.918728 1178 raft_consensus.cc:357] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.918949 1178 raft_consensus.cc:383] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.919013 1178 raft_consensus.cc:738] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.819617 847 leader_election.cc:304] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.871129 2983 raft_consensus.cc:491] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.872308 3011 tablet.cc:1721] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:47.919263 2983 raft_consensus.cc:513] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:47.919922 2983 leader_election.cc:290] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.920271 2983 raft_consensus.cc:2798] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.920382 2983 raft_consensus.cc:695] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:47.920635 2983 consensus_queue.cc:237] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.921653 2923 raft_consensus.cc:513] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.922490 2923 leader_election.cc:290] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:47.924899 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 105310us
I20250112 20:51:47.925868 2975 consensus_queue.cc:1035] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.884734 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 891) took 2970 ms. Trace:
I20250112 20:51:47.926198 1186 rpcz_store.cc:276] 0112 20:51:44.913809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.914429 (+ 620us) service_pool.cc:225] Handling call
0112 20:51:47.551515 (+2637086us) ts_tablet_manager.cc:1145] Deleting tablet 49b8cc99742f4f9ba68eb1aa17cab248
0112 20:51:47.551527 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.557348 (+ 5821us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x616000280954. stack: 00007fd9ea48afd6 00007fd9ea48ad90 00007fd9e946a48d 000000000096d544 000000000096d50d 000000000096ceda 00007fd9f5d2ded0 00007fd9f7022463 00007fd9f9ac6eb7 00007fd9f9af9584 00007fd9f9ada690 00007fd9f9ada41d 0000000000a71ecc 00007fd9ea4d1b64 00007fd9ea4d518b 00007fd9ea4d5008
0112 20:51:47.655954 (+ 98606us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.750279 (+ 94325us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.884711 (+134432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12404992,"tablet-delete.queue_time_us":2636992}
I20250112 20:51:47.926734 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 893) took 3012 ms. Trace:
I20250112 20:51:47.926815 1186 rpcz_store.cc:276] 0112 20:51:44.913980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.920837 (+ 6857us) service_pool.cc:225] Handling call
0112 20:51:47.926599 (+3005762us) ts_tablet_manager.cc:1145] Deleting tablet 54ab0ae10f09469fa4f913e049002481
0112 20:51:47.926615 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.926704 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005589}
I20250112 20:51:47.927109 1186 tablet_replica.cc:331] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:47.927070 847 leader_election.cc:343] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:47.927175 847 leader_election.cc:304] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:47.927332 1186 raft_consensus.cc:2238] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:47.927675 2975 raft_consensus.cc:2743] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.927927 1186 raft_consensus.cc:2267] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.928488 2975 consensus_queue.cc:1035] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.933087 1045 raft_consensus.cc:3054] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.934263 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.934849 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.934999 913 raft_consensus.cc:2463] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.935294 983 leader_election.cc:304] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.935596 2963 raft_consensus.cc:2798] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.935657 2963 raft_consensus.cc:491] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.935706 2963 raft_consensus.cc:3054] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.936132 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9989f3dc61df4f75b9bdb85ba3ca3f37" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.936283 1046 raft_consensus.cc:2463] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:47.834728 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c07855f83f484e62b400775f71925857 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.936995 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07855f83f484e62b400775f71925857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.938028 2963 raft_consensus.cc:513] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.938582 1040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:47.938699 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61964ec077094c018c5a03ad7699c9c6 consensus state has no leader
I20250112 20:51:47.938970 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.939117 913 raft_consensus.cc:3054] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.939576 2964 leader_election.cc:290] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.834787 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9a242579444a1aec77dd3b0e58e8f (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.941116 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a9a242579444a1aec77dd3b0e58e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.941524 913 raft_consensus.cc:2463] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.941998 983 leader_election.cc:304] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.942418 2963 leader_election.cc:290] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.943243 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 49b8cc99742f4f9ba68eb1aa17cab248 (table default.Table_25113f809e324283a5b53261123398f1 [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]) successfully deleted
W20250112 20:51:47.943450 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 54ab0ae10f09469fa4f913e049002481 because the tablet was not found. No further retry: Not found: Tablet not found: 54ab0ae10f09469fa4f913e049002481
I20250112 20:51:47.835048 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 78b166a5954945d0a6bb167ba68cc19a (DEFAULT_TABLE table=default.Table_b70a9a41263b4b2ea19447b0d87b0efc [id=46cd8905ee754e399c75499a2f25f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.835091 1018 tablet_service.cc:1467] Processing CreateTablet for tablet fac60c89bca348dbb8eec04ed74feacc (DEFAULT_TABLE table=default.Table_106af948e5aa4b87830e03feedeae902 [id=e00660b6c95e4309b6ca71e58402eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.835208 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:47.835361 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 6181b166b5cb429da3ebba23cf267942 (DEFAULT_TABLE table=default.Table_638524600a3c447b90e0c84bf8d929c5 [id=d8d3bdf1a89048d394b6cd0456735f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.836328 1047 raft_consensus.cc:3054] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.945037 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b166a5954945d0a6bb167ba68cc19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.945416 1178 consensus_queue.cc:260] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.946967 2975 raft_consensus.cc:491] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.947068 2975 raft_consensus.cc:513] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.947351 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d24bccf682483aa9b273fc1ce847da" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.947530 1046 raft_consensus.cc:2388] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.947582 2975 leader_election.cc:290] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:47.947882 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6181b166b5cb429da3ebba23cf267942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.948551 1178 ts_tablet_manager.cc:1428] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250112 20:51:47.948805 1178 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:47.950129 1178 tablet_bootstrap.cc:654] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.951097 877 tablet_service.cc:1467] Processing CreateTablet for tablet c54516970d424f7cb28c72d8fe52b824 (DEFAULT_TABLE table=default.Table_2ac8449ec8b94e0494cceb60303f0481 [id=3ca64f2a762445789e9a422e02f67cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.951908 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac60c89bca348dbb8eec04ed74feacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.952054 2968 raft_consensus.cc:2798] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.952165 2968 raft_consensus.cc:695] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.952423 2968 consensus_queue.cc:237] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.837874 2926 raft_consensus.cc:2743] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.839205 1163 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.953284 1163 ts_tablet_manager.cc:1397] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.000s
I20250112 20:51:47.847402 2563 raft_consensus.cc:2743] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.887535 1041 raft_consensus.cc:2463] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.954201 1163 raft_consensus.cc:357] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.954418 1163 raft_consensus.cc:383] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.954489 1163 raft_consensus.cc:738] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.954723 1163 consensus_queue.cc:260] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.890295 1049 raft_consensus.cc:2463] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.892486 2787 raft_consensus.cc:513] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.955397 1163 ts_tablet_manager.cc:1428] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:47.955611 2787 leader_election.cc:290] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.955693 1163 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.956166 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.956359 913 raft_consensus.cc:2388] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:47.956979 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54516970d424f7cb28c72d8fe52b824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.958882 1163 tablet_bootstrap.cc:654] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.963970 1186 ts_tablet_manager.cc:1905] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.964884 2787 raft_consensus.cc:491] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.965019 2787 raft_consensus.cc:513] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.965610 2787 leader_election.cc:290] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.966392 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.966639 913 raft_consensus.cc:2463] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.967657 983 leader_election.cc:304] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.968050 2787 raft_consensus.cc:2798] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.968133 2787 raft_consensus.cc:491] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.968372 2787 raft_consensus.cc:3054] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.893466 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 876) took 2329 ms. Trace:
I20250112 20:51:47.899717 2933 raft_consensus.cc:2743] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.969276 2447 rpcz_store.cc:276] 0112 20:51:45.563844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.565654 (+ 1810us) service_pool.cc:225] Handling call
0112 20:51:47.501830 (+1936176us) ts_tablet_manager.cc:1145] Deleting tablet 921ae136d5e2438e950b96f156f23da3
0112 20:51:47.501839 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.577763 (+ 75924us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.727640 (+149877us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.893448 (+165808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935802}
I20250112 20:51:47.969702 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 877) took 2400 ms. Trace:
I20250112 20:51:47.969978 2447 rpcz_store.cc:276] 0112 20:51:45.569526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.576348 (+ 6822us) service_pool.cc:225] Handling call
0112 20:51:47.969595 (+2393247us) ts_tablet_manager.cc:1145] Deleting tablet fc1335522a384eb98163a2539630b93e
0112 20:51:47.969604 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.969688 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393021}
I20250112 20:51:47.970299 2447 tablet_replica.cc:331] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:47.970507 2447 raft_consensus.cc:2238] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:47.970681 2447 raft_consensus.cc:2267] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.971083 2787 raft_consensus.cc:513] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.971221 1186 ts_tablet_manager.cc:1918] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:47.971349 2447 ts_tablet_manager.cc:1905] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:47.971359 1186 log.cc:1198] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e9f669a739d94f99a5b7e9ccccd29545
I20250112 20:51:47.971755 1186 ts_tablet_manager.cc:1939] T e9f669a739d94f99a5b7e9ccccd29545 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:47.972666 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 892) took 3058 ms. Trace:
I20250112 20:51:47.885022 911 raft_consensus.cc:2463] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.891328 908 raft_consensus.cc:2463] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:47.973760 1163 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.973960 1163 ts_tablet_manager.cc:1397] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20250112 20:51:47.974589 1163 raft_consensus.cc:357] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.974769 1163 raft_consensus.cc:383] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.974819 1163 raft_consensus.cc:738] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.974982 1163 consensus_queue.cc:260] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.975183 2787 leader_election.cc:290] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.975229 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.975389 911 raft_consensus.cc:3054] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.975436 1163 ts_tablet_manager.cc:1428] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:47.975718 1163 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:47.977342 1163 tablet_bootstrap.cc:654] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.977960 911 raft_consensus.cc:2463] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.978389 983 leader_election.cc:304] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.978688 2787 raft_consensus.cc:2798] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.978889 2787 raft_consensus.cc:695] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:47.979143 2787 consensus_queue.cc:237] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.917635 2965 raft_consensus.cc:491] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.931402 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:47.931718 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:47.980557 1044 raft_consensus.cc:2388] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:47.981077 2965 raft_consensus.cc:513] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.983279 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:47.983505 911 raft_consensus.cc:2463] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:47.983845 1186 rpcz_store.cc:276] 0112 20:51:44.913902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.929566 (+ 15664us) service_pool.cc:225] Handling call
0112 20:51:47.927059 (+2997493us) ts_tablet_manager.cc:1145] Deleting tablet e9f669a739d94f99a5b7e9ccccd29545
0112 20:51:47.927069 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.968239 (+ 41170us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.971082 (+ 2843us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:47.972644 (+ 1562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":355,"tablet-delete.queue_time_us":2997343}
I20250112 20:51:47.986053 983 leader_election.cc:304] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.986389 2933 raft_consensus.cc:2798] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.986474 2933 raft_consensus.cc:491] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.986546 2933 raft_consensus.cc:3054] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.988265 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 896) took 3018 ms. Trace:
I20250112 20:51:47.988356 1186 rpcz_store.cc:276] 0112 20:51:44.969383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.969624 (+ 241us) service_pool.cc:225] Handling call
0112 20:51:47.988087 (+3018463us) ts_tablet_manager.cc:1145] Deleting tablet 3dc719cd84124082b782016011e37c8b
0112 20:51:47.988101 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.988245 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3018252}
I20250112 20:51:47.988704 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 897) took 3006 ms. Trace:
I20250112 20:51:47.988772 1186 rpcz_store.cc:276] 0112 20:51:44.982051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:44.982398 (+ 347us) service_pool.cc:225] Handling call
0112 20:51:47.988606 (+3006208us) ts_tablet_manager.cc:1145] Deleting tablet cc9d5d182827465abd3799d3ad9a6484
0112 20:51:47.988616 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.988691 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005977}
I20250112 20:51:47.989818 2933 raft_consensus.cc:513] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.991106 2933 leader_election.cc:290] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:47.993029 1163 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:47.993160 1163 ts_tablet_manager.cc:1397] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20250112 20:51:47.994819 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:47.995448 911 raft_consensus.cc:3054] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.996516 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c39266c8bbc54b86921b055275242fc8" } modify_external_catalogs: true
I20250112 20:51:47.996737 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c39266c8bbc54b86921b055275242fc8" } modify_external_catalogs: true
I20250112 20:51:47.998225 911 raft_consensus.cc:2463] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:47.998296 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d8d06d1114e0487d8f3097053759ceb8" } modify_external_catalogs: true
I20250112 20:51:47.998481 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d8d06d1114e0487d8f3097053759ceb8" } modify_external_catalogs: true
I20250112 20:51:47.998739 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 898) took 2938 ms. Trace:
I20250112 20:51:47.998817 1186 rpcz_store.cc:276] 0112 20:51:45.060212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.060349 (+ 137us) service_pool.cc:225] Handling call
0112 20:51:47.998627 (+2938278us) ts_tablet_manager.cc:1145] Deleting tablet d80791d8582446b89adf212d3d6507eb
0112 20:51:47.998636 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.998723 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2938006}
I20250112 20:51:47.998919 983 leader_election.cc:304] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:47.998893 1163 raft_consensus.cc:357] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.999054 1163 raft_consensus.cc:383] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.999074 1186 tablet_replica.cc:331] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:47.999109 1163 raft_consensus.cc:738] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.999270 1186 raft_consensus.cc:2238] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:47.999312 2563 raft_consensus.cc:491] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.999289 1163 consensus_queue.cc:260] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.999449 1186 raft_consensus.cc:2267] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:47.999415 2563 raft_consensus.cc:513] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:47.999748 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ef26f7675c9640a7ac1673df48f024ac" } new_table_name: "default.Table_8b22dc96a929414faa15f9131ffc49b1" modify_external_catalogs: true
I20250112 20:51:47.999835 1163 ts_tablet_manager.cc:1428] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250112 20:51:48.000016 2563 leader_election.cc:290] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.000140 1163 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:48.000147 2563 raft_consensus.cc:2798] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.000897 2563 raft_consensus.cc:695] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.001122 2563 consensus_queue.cc:237] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.001103 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_691a39c63d3a4966914ce5e983c87d47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.001334 2975 raft_consensus.cc:491] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.001474 2975 raft_consensus.cc:513] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:48.001619 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691a39c63d3a4966914ce5e983c87d47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.002008 2975 leader_election.cc:290] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:48.002278 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.002517 911 raft_consensus.cc:2463] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.002768 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a" } modify_external_catalogs: true
I20250112 20:51:48.002938 983 leader_election.cc:304] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.002947 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c6c9942d56944b99a7b1fc6e7c3ec49a" } modify_external_catalogs: true
I20250112 20:51:48.004112 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_48039b63c7ee45558ce7a911e805f60a" } new_table_name: "default.Table_b3d4c836e34b461690df48ea31e4617a" modify_external_catalogs: true
I20250112 20:51:48.005582 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4dc07bdc6b32422eb695a7f4f5710fce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.005363 2933 raft_consensus.cc:2798] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.005929 2933 raft_consensus.cc:491] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.005986 2933 raft_consensus.cc:3054] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:48.006042 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc07bdc6b32422eb695a7f4f5710fce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.008884 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b440da020d947dda2553d65dd6a4bdf with delete_type TABLET_DATA_DELETED (replaced by tablet 2e163a5b99a44d34bf59ada02c2bba99) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.008952 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 226ea7f38f1445c598676715a3036227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.011011 2933 raft_consensus.cc:513] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.011938 2933 leader_election.cc:290] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.012213 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.005352 1163 tablet_bootstrap.cc:654] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.012403 911 raft_consensus.cc:3054] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.014205 1164 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.014356 1164 ts_tablet_manager.cc:1397] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.194s user 0.038s sys 0.030s
I20250112 20:51:48.014952 1164 raft_consensus.cc:357] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.015209 1164 raft_consensus.cc:383] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.015301 1164 raft_consensus.cc:738] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.015470 911 raft_consensus.cc:2463] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.015547 1164 consensus_queue.cc:260] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.015918 1185 ts_tablet_manager.cc:1918] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.015978 983 leader_election.cc:304] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.016100 1185 log.cc:1198] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/c2664a2e98f74f73ab2dc0adaa2335d3
I20250112 20:51:48.016237 2933 raft_consensus.cc:2798] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.016463 1185 ts_tablet_manager.cc:1939] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.016831 1164 ts_tablet_manager.cc:1428] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:48.017088 1164 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.017140 739 tablet_service.cc:1467] Processing CreateTablet for tablet 78b166a5954945d0a6bb167ba68cc19a (DEFAULT_TABLE table=default.Table_b70a9a41263b4b2ea19447b0d87b0efc [id=46cd8905ee754e399c75499a2f25f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.017195 447 catalog_manager.cc:5526] T 8056e2e94fa94fc0b7f03fede018dd8e P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.017429 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.017884 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6181b166b5cb429da3ebba23cf267942 (DEFAULT_TABLE table=default.Table_638524600a3c447b90e0c84bf8d929c5 [id=d8d3bdf1a89048d394b6cd0456735f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.018038 447 catalog_manager.cc:5526] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.018414 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:48.018474 2933 raft_consensus.cc:695] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.017621 754 tablet_service.cc:1467] Processing CreateTablet for tablet fbcda995c4104de78fb694e6842a4796 (DEFAULT_TABLE table=default.Table_b5c457468d7d49078cb67c543721f334 [id=9102c8e30e7047139db701a785f7a30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.018632 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:48.018708 2933 consensus_queue.cc:237] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.018831 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:48.018843 741 tablet_service.cc:1467] Processing CreateTablet for tablet cfb833659c23431daaab008dcdf1de43 (DEFAULT_TABLE table=default.Table_f81091f6a28b413a91f2af4ff8f05df5 [id=37c397b65b7a434e954f9b2cba10b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.019022 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
I20250112 20:51:48.019277 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:48.019373 753 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9a242579444a1aec77dd3b0e58e8f (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.019454 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:48.019601 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:48.019794 447 catalog_manager.cc:5526] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.020284 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:48.020452 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:48.021284 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 89f9b46da06a476ab9f2a93ca74ff736 (table default.Table_a6c9111b483148718ccd2bfb2a54698b [id=b3b49691fa8a41dda2abad8d2f59e423]) successfully deleted
W20250112 20:51:48.021504 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 015f54bdae6b4333963f9e085c5189e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015f54bdae6b4333963f9e085c5189e8 already in progress: creating tablet
W20250112 20:51:48.021787 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 0d072ebae78e40c48e158eb65775476c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d072ebae78e40c48e158eb65775476c already in progress: creating tablet
I20250112 20:51:48.021893 742 tablet_service.cc:1467] Processing CreateTablet for tablet 37504fc56bc248ae86b6f16feb0ce358 (DEFAULT_TABLE table=default.Table_6d10488a4e9d43dea17391673fb9db7b [id=b8356b9c0b9a45b1b92475664cf069db]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:48.022293 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3f6ad2ac50c745448031bdf94e0eea59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f6ad2ac50c745448031bdf94e0eea59 already in progress: creating tablet
W20250112 20:51:48.022495 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 69012f97aac1441095aa3bb1548de9cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69012f97aac1441095aa3bb1548de9cd already in progress: creating tablet
W20250112 20:51:48.023694 440 catalog_manager.cc:4945] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): alter failed for tablet 64e1c32cab2c4b438d3ff4a7e674feeb (table default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]),no further retry: Not found: Tablet not found: 64e1c32cab2c4b438d3ff4a7e674feeb
I20250112 20:51:48.027352 752 tablet_service.cc:1467] Processing CreateTablet for tablet fac60c89bca348dbb8eec04ed74feacc (DEFAULT_TABLE table=default.Table_106af948e5aa4b87830e03feedeae902 [id=e00660b6c95e4309b6ca71e58402eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.027851 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6181b166b5cb429da3ebba23cf267942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.029631 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 910) took 2544 ms. Trace:
I20250112 20:51:48.036274 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcda995c4104de78fb694e6842a4796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.037946 1185 rpcz_store.cc:276] 0112 20:51:45.475547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.477336 (+ 1789us) service_pool.cc:225] Handling call
0112 20:51:47.860599 (+2383263us) ts_tablet_manager.cc:1145] Deleting tablet c2664a2e98f74f73ab2dc0adaa2335d3
0112 20:51:47.860605 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:47.914144 (+ 53539us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.015759 (+101615us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.019946 (+ 4187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2383114}
I20250112 20:51:48.038631 2954 raft_consensus.cc:491] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.038667 747 tablet_service.cc:1467] Processing CreateTablet for tablet 754641e1c24b45faaabfb6c001d4647a (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.038648 738 tablet_service.cc:1467] Processing CreateTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b (DEFAULT_TABLE table=default.Table_da12b2e64bd640d9b17b825b8653b314 [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.039018 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb833659c23431daaab008dcdf1de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.038779 2954 raft_consensus.cc:513] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.039592 2954 leader_election.cc:290] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.042922 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b166a5954945d0a6bb167ba68cc19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.044396 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet c2664a2e98f74f73ab2dc0adaa2335d3 (table default.Table_42c5efdd34e643cba03f44dc9c276e53 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]) successfully deleted
I20250112 20:51:48.044950 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37504fc56bc248ae86b6f16feb0ce358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.045621 2998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250112 20:51:48.048846 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5808d790defa44d7b6067522207c1022 consensus state has no leader
I20250112 20:51:48.051045 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d7aa33578d9f44e68d9056a143371461
I20250112 20:51:48.051342 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f81091f6a28b413a91f2af4ff8f05df5" } modify_external_catalogs: true
I20250112 20:51:48.051501 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f81091f6a28b413a91f2af4ff8f05df5" } modify_external_catalogs: true
I20250112 20:51:48.056861 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a9a242579444a1aec77dd3b0e58e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.058760 1186 ts_tablet_manager.cc:1905] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.061017 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac60c89bca348dbb8eec04ed74feacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.062494 2757 fs_manager.cc:730] Time spent opening block manager: real 0.328s user 0.000s sys 0.013s
I20250112 20:51:48.062686 2757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
uuid: "b985fd656ce44131bc91ccfeb79587c5"
format_stamp: "Formatted at 2025-01-12 20:51:21 on dist-test-slave-4dz2"
I20250112 20:51:48.063899 2757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:48.082273 744 tablet_service.cc:1467] Processing CreateTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 (DEFAULT_TABLE table=default.Table_826881febc214de081501f6ecb406972 [id=fc4ac00547a44ec7bc6f45b69b4c8673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.082825 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff6b3dbc614785a5db4b4185fe0945. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:48.084391 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f669f2f5bed64a6280189cdee98bc64d (table default.Table_162179129dc94078b87f0bf9820e6253 [id=54217c15618341679237c44c98af621d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f669f2f5bed64a6280189cdee98bc64d consensus state has no leader
I20250112 20:51:48.085093 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 911) took 2599 ms. Trace:
I20250112 20:51:48.085211 1185 rpcz_store.cc:276] 0112 20:51:45.485348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.485410 (+ 62us) service_pool.cc:225] Handling call
0112 20:51:48.038258 (+2552848us) ts_tablet_manager.cc:1145] Deleting tablet 3b0786594542416cb67c4112921943b3
0112 20:51:48.084895 (+ 46637us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.085049 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2552652}
I20250112 20:51:48.085708 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754641e1c24b45faaabfb6c001d4647a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.088982 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f6a33e6c95485db80f69fab60d5c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.092969 1185 tablet_replica.cc:331] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.093199 1185 raft_consensus.cc:2238] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.093515 1185 raft_consensus.cc:2267] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.094135 1185 ts_tablet_manager.cc:1905] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:48.101995 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3b0786594542416cb67c4112921943b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0786594542416cb67c4112921943b3 already in progress: creating tablet
I20250112 20:51:48.104313 2933 raft_consensus.cc:491] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.104485 2933 raft_consensus.cc:513] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.105322 2933 leader_election.cc:290] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.105443 2963 raft_consensus.cc:491] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.105597 2963 raft_consensus.cc:513] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.106181 2963 leader_election.cc:290] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.106133 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.106367 911 raft_consensus.cc:2463] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.106896 983 leader_election.cc:304] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.107277 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.107494 911 raft_consensus.cc:2388] T fa2f3389061143509354f37ae93ef0c7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:48.108314 2933 raft_consensus.cc:2798] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.108405 2933 raft_consensus.cc:491] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.108476 2933 raft_consensus.cc:3054] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.111752 2933 raft_consensus.cc:513] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.112262 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 71743c73b6f24da9b14a3d4148117813
I20250112 20:51:48.112399 2933 leader_election.cc:290] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.113020 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.113184 911 raft_consensus.cc:3054] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.113050 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_040d82aa49ea41d8b4bdfe798a1ae64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.113256 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 71743c73b6f24da9b14a3d4148117813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.113418 1164 tablet_bootstrap.cc:654] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:48.113623 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_040d82aa49ea41d8b4bdfe798a1ae64c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.115381 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7e689916a41242d4bc7eec11c33be6b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.115820 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e689916a41242d4bc7eec11c33be6b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.116364 911 raft_consensus.cc:2463] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
W20250112 20:51:48.117300 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d44c14aead8249a291b194f228c5ca2b consensus state has no leader
I20250112 20:51:48.117365 983 leader_election.cc:304] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.117661 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a51f9960ef424b2e9ccca224a6a1654e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.118156 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a51f9960ef424b2e9ccca224a6a1654e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.118410 2933 raft_consensus.cc:2798] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.118553 2933 raft_consensus.cc:695] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.118845 2933 consensus_queue.cc:237] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.129841 2965 leader_election.cc:290] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.132200 1164 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.132359 1164 ts_tablet_manager.cc:1397] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.005s
I20250112 20:51:48.132500 1186 ts_tablet_manager.cc:1918] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.132602 1186 log.cc:1198] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1d97ac93467a4a7a8ae44d52719f191c
I20250112 20:51:48.133077 1164 raft_consensus.cc:357] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.133273 1164 raft_consensus.cc:383] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.133330 1164 raft_consensus.cc:738] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.134268 1164 consensus_queue.cc:260] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.136878 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557fa6445eee481dbc4076f67fe63466" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.137044 908 raft_consensus.cc:3054] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.137638 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.137862 775 raft_consensus.cc:2463] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.138284 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 2c8c7784e5534c2c91f124dffd24d5c5
I20250112 20:51:48.138618 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.138798 775 raft_consensus.cc:3054] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.139628 908 raft_consensus.cc:2463] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.139827 1164 ts_tablet_manager.cc:1428] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:48.139904 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.140125 913 raft_consensus.cc:2388] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.140157 1164 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.140395 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.140538 907 raft_consensus.cc:3054] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.141690 775 raft_consensus.cc:2463] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.142210 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.142375 775 raft_consensus.cc:3054] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.143028 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c0ad7aebc1e245b2bcd5c4af8c8211c6" } modify_external_catalogs: true
I20250112 20:51:48.143214 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c0ad7aebc1e245b2bcd5c4af8c8211c6" } modify_external_catalogs: true
I20250112 20:51:48.145100 775 raft_consensus.cc:2463] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.145977 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.144107 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.149560 909 raft_consensus.cc:3054] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:48.150703 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66fa87d8f0c64f7aa616882fe399ef0e consensus state has no leader
I20250112 20:51:48.152292 909 raft_consensus.cc:2463] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.144182 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.152619 906 raft_consensus.cc:3054] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.153716 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_799f579511f14383a50065deef83f536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.154209 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_799f579511f14383a50065deef83f536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.155261 906 raft_consensus.cc:2463] T 1b1565c7487c428f8f963aca406263fe P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.144275 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.155642 912 raft_consensus.cc:3054] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.157838 912 raft_consensus.cc:2463] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.148609 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.158407 911 raft_consensus.cc:2388] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.149035 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.158723 913 raft_consensus.cc:2388] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.158922 1185 ts_tablet_manager.cc:1918] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.159042 1185 log.cc:1198] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b4ef01b41b6140d6bdad07bc79dd7dc5
I20250112 20:51:48.159466 1185 ts_tablet_manager.cc:1939] T b4ef01b41b6140d6bdad07bc79dd7dc5 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.160205 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 912) took 2673 ms. Trace:
I20250112 20:51:48.160281 1185 rpcz_store.cc:276] 0112 20:51:45.487027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.487079 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:48.092896 (+2605817us) ts_tablet_manager.cc:1145] Deleting tablet b4ef01b41b6140d6bdad07bc79dd7dc5
0112 20:51:48.092905 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.117450 (+ 24545us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.158793 (+ 41343us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.160185 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605644}
I20250112 20:51:48.143838 907 raft_consensus.cc:2463] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.160719 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (table default.Table_0787fb4223774e1889d1d09fb80f87f5 [id=fb7c117897c24636b7e2dd8109e782d2]) successfully deleted
I20250112 20:51:48.161749 1185 tablet_replica.cc:331] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.163039 877 tablet_service.cc:1467] Processing CreateTablet for tablet ffe14c8f0406451bae99014c98c6cae1 (DEFAULT_TABLE table=default.Table_d9a46ff1902a48529ee58d8c36e0d9e0 [id=2c0c563d63d147b2b7e791413c988823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.163478 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe14c8f0406451bae99014c98c6cae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.164019 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_737195573b924b53b18c84b5b0f7f44c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.164502 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737195573b924b53b18c84b5b0f7f44c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.173425 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc545822e98460e815df4e6cad455b4 (DEFAULT_TABLE table=default.Table_96d7128d31fe403abd7f96408718800c [id=2b3b843a4c534104acefa550d9af6545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.173810 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc545822e98460e815df4e6cad455b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.181196 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.181989 1018 tablet_service.cc:1467] Processing CreateTablet for tablet fbcda995c4104de78fb694e6842a4796 (DEFAULT_TABLE table=default.Table_b5c457468d7d49078cb67c543721f334 [id=9102c8e30e7047139db701a785f7a30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.182114 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cfb833659c23431daaab008dcdf1de43 (DEFAULT_TABLE table=default.Table_f81091f6a28b413a91f2af4ff8f05df5 [id=37c397b65b7a434e954f9b2cba10b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.182312 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 71743c73b6f24da9b14a3d4148117813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.182646 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 37504fc56bc248ae86b6f16feb0ce358 (DEFAULT_TABLE table=default.Table_6d10488a4e9d43dea17391673fb9db7b [id=b8356b9c0b9a45b1b92475664cf069db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.182806 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b (DEFAULT_TABLE table=default.Table_da12b2e64bd640d9b17b825b8653b314 [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.182650 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 754641e1c24b45faaabfb6c001d4647a (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.183054 1010 tablet_service.cc:1467] Processing CreateTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 (DEFAULT_TABLE table=default.Table_826881febc214de081501f6ecb406972 [id=fc4ac00547a44ec7bc6f45b69b4c8673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.182652 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.183107 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcda995c4104de78fb694e6842a4796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.183701 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f6a33e6c95485db80f69fab60d5c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.184494 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 921ae136d5e2438e950b96f156f23da3 (table default.Table_a18054e19f524ebd88965aa3727a35f8 [id=da834cd169b04664b22729cfadf3b5e6]) successfully deleted
W20250112 20:51:48.184725 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet fc1335522a384eb98163a2539630b93e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1335522a384eb98163a2539630b93e already in progress: creating tablet
I20250112 20:51:48.184948 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37504fc56bc248ae86b6f16feb0ce358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.186374 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff6b3dbc614785a5db4b4185fe0945. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.191159 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754641e1c24b45faaabfb6c001d4647a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.193018 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb833659c23431daaab008dcdf1de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.196141 1186 ts_tablet_manager.cc:1939] T 1d97ac93467a4a7a8ae44d52719f191c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.204130 884 tablet_service.cc:1467] Processing CreateTablet for tablet bdcd45226ab44353b5574bfd4dae7f9a (DEFAULT_TABLE table=default.Table_132455edcc094422bdf769a1c836cf16 [id=db8fd22edd3b4d4fa88e1f6e1273d666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.204543 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcd45226ab44353b5574bfd4dae7f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.210678 878 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.211014 884 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.211210 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29316b9d2441bca97f56bdb0e56f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.211478 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.211769 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb9c2e6f37f4184a106aed05bae7924 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e8808d62d4b7da88bffe7d7569568) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.212247 889 tablet_service.cc:1467] Processing CreateTablet for tablet f669f2f5bed64a6280189cdee98bc64d (DEFAULT_TABLE table=default.Table_d95d780f503949e3b24983e2c1202f02 [id=54217c15618341679237c44c98af621d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.212617 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f669f2f5bed64a6280189cdee98bc64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.216902 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 9705feb15f384b5185935eacc521d0e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1596ecc13e4a4de58315a28d04697a1b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.217270 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9572179a6f45759214c6c39f22b2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.217593 878 tablet_service.cc:1514] Processing DeleteTablet for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:46 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.217898 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e2c14353c4a8cb7242d75bb3ba673 with delete_type TABLET_DATA_DELETED (replaced by tablet 08b293868ae546c88f393b17e51147d0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.218228 878 tablet_service.cc:1514] Processing DeleteTablet for tablet b062c1d67d4443eabe8ac1d8a832c243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.218578 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 226ea7f38f1445c598676715a3036227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.219341 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b440da020d947dda2553d65dd6a4bdf with delete_type TABLET_DATA_DELETED (replaced by tablet 2e163a5b99a44d34bf59ada02c2bba99) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.219926 878 tablet_service.cc:1467] Processing CreateTablet for tablet c07855f83f484e62b400775f71925857 (DEFAULT_TABLE table=default.Table_a06a212d24b846debb4ec624d3a1f24d [id=af476982644a412fb7af5a4cf44a7846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.220245 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cfb833659c23431daaab008dcdf1de43
I20250112 20:51:48.220301 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07855f83f484e62b400775f71925857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.221041 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_62da44136d7d4f6dbc73e917cf24ad6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.221644 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62da44136d7d4f6dbc73e917cf24ad6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.225405 446 catalog_manager.cc:5526] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.225854 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:48.226207 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]). Expected version 1 got 0
I20250112 20:51:48.226408 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
I20250112 20:51:48.226902 2787 raft_consensus.cc:491] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.227030 2787 raft_consensus.cc:513] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.227113 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:48.227636 2787 leader_election.cc:290] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.227344 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet b930069b6f1b49439dbc0a7abd3fba72 (table default.Table_f2b30f50ce0843e2861e9bae518ea27b [id=61af9ec286544b919c9f053b647bc6fe]). Expected version 1 got 0
I20250112 20:51:48.228245 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:48.228614 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:48.229116 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_f2b30f50ce0843e2861e9bae518ea27b [id=61af9ec286544b919c9f053b647bc6fe] alter complete (version 1)
I20250112 20:51:48.229529 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.229737 907 raft_consensus.cc:2463] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.231370 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d47332a374d74462aa73b72ac64269bf" } new_table_name: "default.Table_72096e102a794d418cb83a674ddaeab6" modify_external_catalogs: true
I20250112 20:51:48.234484 983 leader_election.cc:304] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.234752 2787 raft_consensus.cc:2798] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.234814 2787 raft_consensus.cc:491] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.234903 2787 raft_consensus.cc:3054] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.235810 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd34e829d0645ea80daaff5d27cb266" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.236311 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924082716c1e4d9ea6fbe97ae93232bd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.236481 1039 raft_consensus.cc:3054] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.236913 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557fa6445eee481dbc4076f67fe63466" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.237064 1045 raft_consensus.cc:3054] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.239394 1039 raft_consensus.cc:2463] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.239490 1045 raft_consensus.cc:2463] T 557fa6445eee481dbc4076f67fe63466 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.240540 884 tablet_service.cc:1467] Processing CreateTablet for tablet fac60c89bca348dbb8eec04ed74feacc (DEFAULT_TABLE table=default.Table_106af948e5aa4b87830e03feedeae902 [id=e00660b6c95e4309b6ca71e58402eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.241027 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac60c89bca348dbb8eec04ed74feacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.242643 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e45ad07d53b94b6f803b6b694656de77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.243328 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e45ad07d53b94b6f803b6b694656de77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.243372 2787 raft_consensus.cc:513] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.244408 2787 leader_election.cc:290] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.244501 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.244819 907 raft_consensus.cc:3054] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.247426 878 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9a242579444a1aec77dd3b0e58e8f (DEFAULT_TABLE table=default.Table_96a40b39750b413e9567ab229a39f9b6 [id=f4588bf19e114a19a105776e3013bd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.247712 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 900) took 3185 ms. Trace:
I20250112 20:51:48.247785 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a9a242579444a1aec77dd3b0e58e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.247812 1186 rpcz_store.cc:276] 0112 20:51:45.062614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.062659 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:47.999032 (+2936373us) ts_tablet_manager.cc:1145] Deleting tablet 1d97ac93467a4a7a8ae44d52719f191c
0112 20:51:47.999041 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.129033 (+129992us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.132415 (+ 3382us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.247658 (+115243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936179}
I20250112 20:51:48.247949 907 raft_consensus.cc:2463] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.248261 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 901) took 3185 ms. Trace:
I20250112 20:51:48.248435 983 leader_election.cc:304] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.248672 2787 raft_consensus.cc:2798] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.248767 2787 raft_consensus.cc:695] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.249027 2787 consensus_queue.cc:237] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.249639 1178 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.249765 1178 ts_tablet_manager.cc:1397] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.301s user 0.002s sys 0.000s
I20250112 20:51:48.250465 1178 raft_consensus.cc:357] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.250667 1178 raft_consensus.cc:383] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.250739 1178 raft_consensus.cc:738] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.250928 1178 consensus_queue.cc:260] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.251436 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2664a2e98f74f73ab2dc0adaa2335d3" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.251580 1045 raft_consensus.cc:3054] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.252277 1039 raft_consensus.cc:3054] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.254073 1045 raft_consensus.cc:2463] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.260668 2447 ts_tablet_manager.cc:1918] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.260828 2447 log.cc:1198] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:48.261281 2447 ts_tablet_manager.cc:1939] T 4d29316b9d2441bca97f56bdb0e56f0d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:48.261307 1186 rpcz_store.cc:276] 0112 20:51:45.062732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.062769 (+ 37us) service_pool.cc:225] Handling call
0112 20:51:48.248142 (+3185373us) ts_tablet_manager.cc:1145] Deleting tablet 4561956c97824824a55283a8da21929e
0112 20:51:48.248151 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.248216 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185227}
I20250112 20:51:48.261741 1186 tablet_replica.cc:331] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.261988 1186 raft_consensus.cc:2238] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.262805 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6181b166b5cb429da3ebba23cf267942 (DEFAULT_TABLE table=default.Table_638524600a3c447b90e0c84bf8d929c5 [id=d8d3bdf1a89048d394b6cd0456735f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.263226 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6181b166b5cb429da3ebba23cf267942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.273222 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 146994989d6d4ce8a12e1b8677d7feb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.273917 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5b10576bd34af5b02f8d8dcc3759ab" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.274089 1040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1565c7487c428f8f963aca406263fe" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.274160 1045 raft_consensus.cc:3054] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.274276 1040 raft_consensus.cc:3054] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.276680 1045 raft_consensus.cc:2463] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.278923 885 tablet_service.cc:1467] Processing CreateTablet for tablet 78b166a5954945d0a6bb167ba68cc19a (DEFAULT_TABLE table=default.Table_b70a9a41263b4b2ea19447b0d87b0efc [id=46cd8905ee754e399c75499a2f25f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.279498 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78b166a5954945d0a6bb167ba68cc19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.287285 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet db9bab2477a94c169817d9de8aaad1c0
I20250112 20:51:48.291860 888 tablet_service.cc:1467] Processing CreateTablet for tablet 754641e1c24b45faaabfb6c001d4647a (DEFAULT_TABLE table=default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.292289 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754641e1c24b45faaabfb6c001d4647a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.292526 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_46854461640b4fec8b578900ba36d90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.293064 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46854461640b4fec8b578900ba36d90e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.294323 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_96a40b39750b413e9567ab229a39f9b6" } new_table_name: "default.Table_4a8d364e4d444cff9b191f6e2ef71245" modify_external_catalogs: true
I20250112 20:51:48.298938 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_857cd22bec1a484eb38f055272a9a9a8" } new_table_name: "default.Table_8f71e61c496a49e18c699f7850924379" modify_external_catalogs: true
I20250112 20:51:48.169642 2954 raft_consensus.cc:491] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.300688 2954 raft_consensus.cc:513] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.301337 2954 leader_election.cc:290] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.228029 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.228281 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.228495 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.302241 768 raft_consensus.cc:2463] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.228717 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.302495 761 raft_consensus.cc:3054] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.228838 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.305063 774 raft_consensus.cc:2388] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:48.229621 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.305920 761 raft_consensus.cc:2463] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.229993 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f3389061143509354f37ae93ef0c7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.230450 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.231194 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9989f3dc61df4f75b9bdb85ba3ca3f37" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.306936 776 raft_consensus.cc:2463] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:48.231890 3008 tablet.cc:1721] T b930069b6f1b49439dbc0a7abd3fba72 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:48.232596 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.232954 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.233493 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.229912 752 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb833659c23431daaab008dcdf1de43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.289599 744 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.161964 1185 raft_consensus.cc:2238] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.313280 1185 raft_consensus.cc:2267] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.313915 1185 ts_tablet_manager.cc:1905] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.313983 2787 raft_consensus.cc:491] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.314133 2787 raft_consensus.cc:513] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.314666 2787 leader_election.cc:290] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.202520 1164 tablet_bootstrap.cc:654] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.229128 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3a9740415140fe8864d54f477af954" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.315562 760 raft_consensus.cc:2388] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:48.316286 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fbe3a476695f40ec8737a0e3e4ad731f" } modify_external_catalogs: true
I20250112 20:51:48.316453 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fbe3a476695f40ec8737a0e3e4ad731f" } modify_external_catalogs: true
I20250112 20:51:48.316977 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a6b14949b43a4bbd81fd8004f55380bb" } modify_external_catalogs: true
I20250112 20:51:48.317037 877 tablet_service.cc:1467] Processing CreateTablet for tablet 37504fc56bc248ae86b6f16feb0ce358 (DEFAULT_TABLE table=default.Table_6d10488a4e9d43dea17391673fb9db7b [id=b8356b9c0b9a45b1b92475664cf069db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.317165 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a6b14949b43a4bbd81fd8004f55380bb" } modify_external_catalogs: true
I20250112 20:51:48.317436 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37504fc56bc248ae86b6f16feb0ce358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.234139 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807071e53a4f0e8629b1dfa6d26b9d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.318783 764 raft_consensus.cc:2388] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.319120 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.319365 907 raft_consensus.cc:2463] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.319793 983 leader_election.cc:304] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.320291 2787 raft_consensus.cc:2798] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.320374 2787 raft_consensus.cc:491] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.320441 2787 raft_consensus.cc:3054] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.321702 1040 raft_consensus.cc:2463] T 1b1565c7487c428f8f963aca406263fe P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.322978 2787 raft_consensus.cc:513] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.323531 2787 leader_election.cc:290] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.324232 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.324402 907 raft_consensus.cc:3054] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.325346 2787 raft_consensus.cc:491] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.325527 2787 raft_consensus.cc:513] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.326858 907 raft_consensus.cc:2463] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.326905 2787 leader_election.cc:290] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.327258 983 leader_election.cc:304] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.326860 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.327477 913 raft_consensus.cc:2463] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.327533 2787 raft_consensus.cc:2798] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.327872 983 leader_election.cc:304] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.328229 2787 raft_consensus.cc:695] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.328523 2787 consensus_queue.cc:237] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.329617 2968 raft_consensus.cc:2798] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.329751 2968 raft_consensus.cc:491] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.329973 2968 raft_consensus.cc:3054] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.331571 710 leader_election.cc:304] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.331894 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dba3d9315db4b9292288ee62c2244bc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.333511 888 tablet_service.cc:1467] Processing CreateTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 (DEFAULT_TABLE table=default.Table_826881febc214de081501f6ecb406972 [id=fc4ac00547a44ec7bc6f45b69b4c8673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.333979 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff6b3dbc614785a5db4b4185fe0945. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.334371 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_86e4035224cc49598a669f49ad4553e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.335081 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86e4035224cc49598a669f49ad4553e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.335513 2968 raft_consensus.cc:513] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.340240 2968 leader_election.cc:290] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.340301 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:48.340466 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 261fe76c57b14b35957ca43606cfc02f consensus state has no leader
I20250112 20:51:48.340481 913 raft_consensus.cc:3054] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.341607 875 tablet_service.cc:1467] Processing CreateTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b (DEFAULT_TABLE table=default.Table_da12b2e64bd640d9b17b825b8653b314 [id=10e66a500a5349ad8eb2c03fe9acc25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.342051 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01f6a33e6c95485db80f69fab60d5c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.343497 913 raft_consensus.cc:2463] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.346328 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:48.346591 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:48.346843 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet cc90d3722fd14d59b9994655cd30ef2b (table default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757]). Expected version 4 got 0
I20250112 20:51:48.347276 448 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet c11a892641554e4db9bdd558068cac40 (table default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284]). Expected version 1 got 0
I20250112 20:51:48.349686 738 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.350029 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.351270 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:48.351549 449 catalog_manager.cc:5526] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.352049 1164 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.352120 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:48.352160 1164 ts_tablet_manager.cc:1397] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.212s user 0.022s sys 0.005s
I20250112 20:51:48.352325 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:48.352635 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:48.353044 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.355342 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.355547 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:48.360673 752 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.360909 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.366142 2954 raft_consensus.cc:2798] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.366288 2954 raft_consensus.cc:695] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.366567 2954 consensus_queue.cc:237] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.368723 1164 raft_consensus.cc:357] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.368917 1164 raft_consensus.cc:383] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.368965 1164 raft_consensus.cc:738] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.369141 1164 consensus_queue.cc:260] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.369589 1164 ts_tablet_manager.cc:1428] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250112 20:51:48.370559 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e9a8102300fb471f93b5e5ba18c2f532" } new_table_name: "default.Table_89b5b994fdf143a09d82604bbe6f8253" modify_external_catalogs: true
I20250112 20:51:48.371153 1164 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.372395 1164 tablet_bootstrap.cc:654] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.377148 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 71743c73b6f24da9b14a3d4148117813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:47 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:48.377542 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 45981.1us
I20250112 20:51:48.378379 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b2f736731d74cdfb099e7a498bf92a0" } modify_external_catalogs: true
I20250112 20:51:48.378710 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5b2f736731d74cdfb099e7a498bf92a0" } modify_external_catalogs: true
I20250112 20:51:48.382197 710 leader_election.cc:304] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.384109 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8c7784e5534c2c91f124dffd24d5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:48.384225 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9a9a242579444a1aec77dd3b0e58e8f consensus state has no leader
I20250112 20:51:48.385730 1178 ts_tablet_manager.cc:1428] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.136s user 0.002s sys 0.000s
I20250112 20:51:48.386260 2923 raft_consensus.cc:2798] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.386368 2923 raft_consensus.cc:695] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.386569 2923 consensus_queue.cc:237] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.393539 1164 tablet_bootstrap.cc:492] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.393672 1164 ts_tablet_manager.cc:1397] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.003s
I20250112 20:51:48.394263 1164 raft_consensus.cc:357] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.394445 1164 raft_consensus.cc:383] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.394496 1164 raft_consensus.cc:738] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.394686 1164 consensus_queue.cc:260] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.395138 1164 ts_tablet_manager.cc:1428] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 20:51:48.395392 1164 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.396659 1164 tablet_bootstrap.cc:654] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.398267 2886 raft_consensus.cc:491] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.398384 2886 raft_consensus.cc:513] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.398873 2886 leader_election.cc:290] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.399863 710 leader_election.cc:304] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.404264 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8dc3d4d22c54413aad3f7479f38380e5" } new_table_name: "default.Table_783cfde3de2e4786bb0959e831910f91" modify_external_catalogs: true
I20250112 20:51:48.382041 1163 tablet_bootstrap.cc:492] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:48.406176 1163 ts_tablet_manager.cc:1397] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.406s user 0.043s sys 0.084s
I20250112 20:51:48.406961 1163 raft_consensus.cc:357] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.407163 1163 raft_consensus.cc:383] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.407239 1163 raft_consensus.cc:738] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.407497 1163 consensus_queue.cc:260] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.408044 1163 ts_tablet_manager.cc:1428] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:48.408353 1163 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:48.389101 983 leader_election.cc:304] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.400961 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.401094 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ae136d5e2438e950b96f156f23da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.401165 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.409175 2968 raft_consensus.cc:2798] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.401371 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b8cc99742f4f9ba68eb1aa17cab248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.409287 2968 raft_consensus.cc:695] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.400877 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb833659c23431daaab008dcdf1de43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.401329 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.409514 2968 consensus_queue.cc:237] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.410621 2563 consensus_queue.cc:1035] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.410827 1163 tablet_bootstrap.cc:654] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.414598 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3ea9425f3e7b46288c81dc8cdade99b2
W20250112 20:51:48.416028 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05e171717b7d462399b141aed9be3c08 consensus state has no leader
I20250112 20:51:48.419900 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2f77351ffe3d4bb995bee57e885a76de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.420466 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f77351ffe3d4bb995bee57e885a76de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.380795 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 887) took 2600 ms. Trace:
I20250112 20:51:48.421872 2447 rpcz_store.cc:276] 0112 20:51:45.780409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.781652 (+ 1243us) service_pool.cc:225] Handling call
0112 20:51:47.970259 (+2188607us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:47.970267 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.121605 (+151338us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.260540 (+138935us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.380722 (+120182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2166680}
I20250112 20:51:48.422334 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 888) took 2495 ms. Trace:
I20250112 20:51:48.422672 2886 raft_consensus.cc:2798] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.422802 2886 raft_consensus.cc:695] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.423102 2886 consensus_queue.cc:237] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.416106 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.426251 710 leader_election.cc:304] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:48.426419 710 raft_consensus.cc:2681] T c2664a2e98f74f73ab2dc0adaa2335d3 P aefe1db8b19244d891b6bcd8971e9b17: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:48.426569 710 leader_election.cc:304] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.426770 2886 raft_consensus.cc:2798] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.426885 2886 raft_consensus.cc:695] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.427145 2886 consensus_queue.cc:237] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.386142 1186 raft_consensus.cc:2267] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.434008 1186 ts_tablet_manager.cc:1905] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.435364 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 45484f8e48524b86a6f418b88a71ac18
I20250112 20:51:48.388530 2983 raft_consensus.cc:491] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.436898 2983 raft_consensus.cc:513] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.437536 2983 leader_election.cc:290] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:48.401690 874 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb833659c23431daaab008dcdf1de43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.419384 876 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.440655 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_314fc7eb6aff43d7b18dba3be1a7c618" } modify_external_catalogs: true
I20250112 20:51:48.440927 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_314fc7eb6aff43d7b18dba3be1a7c618" } modify_external_catalogs: true
I20250112 20:51:48.442174 1186 ts_tablet_manager.cc:1918] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.442308 1186 log.cc:1198] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/cbc22207e9204b73877c74f61ebba8a8
I20250112 20:51:48.442690 1186 ts_tablet_manager.cc:1939] T cbc22207e9204b73877c74f61ebba8a8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.419787 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.419915 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.420222 892 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.420928 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.443324 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 45484f8e48524b86a6f418b88a71ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.443531 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.444115 2447 rpcz_store.cc:276] 0112 20:51:45.926709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.927203 (+ 494us) service_pool.cc:225] Handling call
0112 20:51:48.422164 (+2494961us) ts_tablet_manager.cc:1145] Deleting tablet be1f12369c894682b7d30de4e37bd466
0112 20:51:48.422175 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.422282 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494721}
I20250112 20:51:48.444952 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 889) took 2515 ms. Trace:
I20250112 20:51:48.445079 2447 rpcz_store.cc:276] 0112 20:51:45.928979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.929038 (+ 59us) service_pool.cc:225] Handling call
0112 20:51:48.444411 (+2515373us) ts_tablet_manager.cc:1145] Deleting tablet a10f7a47089a43d8ae9c2fc16d379740
0112 20:51:48.444421 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.444517 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515146}
I20250112 20:51:48.445446 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 890) took 2507 ms. Trace:
I20250112 20:51:48.445541 2447 rpcz_store.cc:276] 0112 20:51:45.938052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.938103 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:48.445319 (+2507216us) ts_tablet_manager.cc:1145] Deleting tablet 2aef567a7cf64178b53625ac4638048b
0112 20:51:48.445327 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.445408 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506971}
I20250112 20:51:48.445863 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 891) took 2427 ms. Trace:
I20250112 20:51:48.445953 2447 rpcz_store.cc:276] 0112 20:51:46.018371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.018468 (+ 97us) service_pool.cc:225] Handling call
0112 20:51:48.445738 (+2427270us) ts_tablet_manager.cc:1145] Deleting tablet a2dbdbf416f64011b431f5188e0a06c4
0112 20:51:48.445747 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.445824 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427086}
I20250112 20:51:48.446280 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 892) took 2417 ms. Trace:
I20250112 20:51:48.446369 2447 rpcz_store.cc:276] 0112 20:51:46.029250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.029843 (+ 593us) service_pool.cc:225] Handling call
0112 20:51:48.446165 (+2416322us) ts_tablet_manager.cc:1145] Deleting tablet 53518b453d224ca186fcc6b2ffab5f04
0112 20:51:48.446175 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.446239 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416135}
I20250112 20:51:48.446661 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 893) took 2406 ms. Trace:
I20250112 20:51:48.446754 2447 rpcz_store.cc:276] 0112 20:51:46.040174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.040222 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:48.446557 (+2406335us) ts_tablet_manager.cc:1145] Deleting tablet 734734ff9c564f769bea0984a9e0386b
0112 20:51:48.446565 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.446626 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406152}
I20250112 20:51:48.448108 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 894) took 2321 ms. Trace:
I20250112 20:51:48.451789 2447 rpcz_store.cc:276] 0112 20:51:46.125979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.126031 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:48.446975 (+2320944us) ts_tablet_manager.cc:1145] Deleting tablet e92d58058511437d830a8f3cbba1e206
0112 20:51:48.446987 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.447058 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2320764}
I20250112 20:51:48.452188 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 895) took 2026 ms. Trace:
I20250112 20:51:48.452289 2447 rpcz_store.cc:276] 0112 20:51:46.425377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.425426 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:48.452040 (+2026614us) ts_tablet_manager.cc:1145] Deleting tablet c6bdfecd5d9142d5b008c613864c8217
0112 20:51:48.452049 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.452150 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026419}
I20250112 20:51:48.452630 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 896) took 1958 ms. Trace:
I20250112 20:51:48.452733 2447 rpcz_store.cc:276] 0112 20:51:46.494133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.494189 (+ 56us) service_pool.cc:225] Handling call
0112 20:51:48.452514 (+1958325us) ts_tablet_manager.cc:1145] Deleting tablet f0fb6a5146c44edea076a10da58ea1a7
0112 20:51:48.452522 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.452591 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958106}
I20250112 20:51:48.453081 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 897) took 1937 ms. Trace:
I20250112 20:51:48.453192 2447 rpcz_store.cc:276] 0112 20:51:46.515239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.515290 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:48.452962 (+1937672us) ts_tablet_manager.cc:1145] Deleting tablet f424288fe62849edbbd2d38e89331fb7
0112 20:51:48.452970 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.453040 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937464}
I20250112 20:51:48.463258 889 tablet_service.cc:1467] Processing CreateTablet for tablet fbcda995c4104de78fb694e6842a4796 (DEFAULT_TABLE table=default.Table_b5c457468d7d49078cb67c543721f334 [id=9102c8e30e7047139db701a785f7a30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.464720 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 899) took 3402 ms. Trace:
I20250112 20:51:48.464921 1186 rpcz_store.cc:276] 0112 20:51:45.062457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.062634 (+ 177us) service_pool.cc:225] Handling call
0112 20:51:48.261614 (+3198980us) ts_tablet_manager.cc:1145] Deleting tablet cbc22207e9204b73877c74f61ebba8a8
0112 20:51:48.261624 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.438919 (+177295us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.442053 (+ 3134us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.464698 (+ 22645us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123785,"tablet-delete.queue_time_us":3198672}
I20250112 20:51:48.465353 1178 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:48.466086 1186 tablet_replica.cc:331] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.466526 1186 raft_consensus.cc:2238] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.466715 1178 tablet_bootstrap.cc:654] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.466823 1186 raft_consensus.cc:2267] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.467372 1186 ts_tablet_manager.cc:1905] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.386160 884 tablet_service.cc:1467] Processing CreateTablet for tablet cfb833659c23431daaab008dcdf1de43 (DEFAULT_TABLE table=default.Table_f81091f6a28b413a91f2af4ff8f05df5 [id=37c397b65b7a434e954f9b2cba10b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.468333 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb833659c23431daaab008dcdf1de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.470384 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcda995c4104de78fb694e6842a4796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.477850 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 906) took 1769 ms. Trace:
I20250112 20:51:48.477945 2447 rpcz_store.cc:276] 0112 20:51:46.708461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.708580 (+ 119us) service_pool.cc:225] Handling call
0112 20:51:48.477699 (+1769119us) ts_tablet_manager.cc:1145] Deleting tablet 6158b97ccf0a42fdb1a510c5d1e80213
0112 20:51:48.477710 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.477813 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768941}
I20250112 20:51:48.478224 2447 tablet_replica.cc:331] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:48.478428 2447 raft_consensus.cc:2238] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.478627 2447 raft_consensus.cc:2267] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:48.478978 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 52418.9us
I20250112 20:51:48.479243 710 leader_election.cc:304] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.479502 2886 raft_consensus.cc:2798] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.479626 2886 raft_consensus.cc:695] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.479781 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_72b8c68194b84004b32fa74e32d5e26e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.480365 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72b8c68194b84004b32fa74e32d5e26e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.479849 2886 consensus_queue.cc:237] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.467940 1163 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:48.481737 1163 ts_tablet_manager.cc:1397] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.000s
I20250112 20:51:48.482481 1163 raft_consensus.cc:357] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.482682 1163 raft_consensus.cc:383] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.482751 1163 raft_consensus.cc:738] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.483032 1163 consensus_queue.cc:260] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.483542 1163 ts_tablet_manager.cc:1428] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:48.483805 1163 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:48.493032 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493212 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 924082716c1e4d9ea6fbe97ae93232bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493283 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3a9740415140fe8864d54f477af954 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493394 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493646 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d24bccf682483aa9b273fc1ce847da with delete_type TABLET_DATA_DELETED (replaced by tablet 370ad081bf784808b2ab44cad8d80959) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493753 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd34e829d0645ea80daaff5d27cb266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493881 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.494011 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ea9425f3e7b46288c81dc8cdade99b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.493674 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fd47d1c1a4b42bdbff97f619dc95134 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.494089 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6188282bedce4235988a0602b6addcaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.494256 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f807071e53a4f0e8629b1dfa6d26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:48.494637 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6188282bedce4235988a0602b6addcaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.495724 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 45484f8e48524b86a6f418b88a71ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.498950 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e9f669a739d94f99a5b7e9ccccd29545 (table default.Table_0b5b47e1cac64ab19d29a9a39e8b19e7 [id=22271590d9384d558a11cab8e3c71131]) successfully deleted
W20250112 20:51:48.499624 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cc9d5d182827465abd3799d3ad9a6484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9d5d182827465abd3799d3ad9a6484 already in progress: creating tablet
W20250112 20:51:48.499992 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d80791d8582446b89adf212d3d6507eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d80791d8582446b89adf212d3d6507eb already in progress: creating tablet
I20250112 20:51:48.500910 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.503767 710 leader_election.cc:304] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:48.505327 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.505548 907 raft_consensus.cc:2388] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.506095 2886 raft_consensus.cc:2798] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.506369 2886 raft_consensus.cc:695] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:48.507098 2886 consensus_queue.cc:237] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.514725 2447 ts_tablet_manager.cc:1905] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:48.514899 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5e07f13efff44d1a0a4bb8508c16e12 consensus state has no leader
I20250112 20:51:48.518762 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 4d29316b9d2441bca97f56bdb0e56f0d (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
W20250112 20:51:48.519217 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet be1f12369c894682b7d30de4e37bd466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1f12369c894682b7d30de4e37bd466 already in progress: creating tablet
I20250112 20:51:48.520701 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1d97ac93467a4a7a8ae44d52719f191c (table default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]) successfully deleted
W20250112 20:51:48.521386 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4561956c97824824a55283a8da21929e because the tablet was not found. No further retry: Not found: Tablet not found: 4561956c97824824a55283a8da21929e
I20250112 20:51:48.521847 1178 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.521986 1178 ts_tablet_manager.cc:1397] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20250112 20:51:48.522687 1178 raft_consensus.cc:357] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.522866 1178 raft_consensus.cc:383] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.522915 1178 raft_consensus.cc:738] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.523137 1178 consensus_queue.cc:260] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.523784 847 leader_election.cc:304] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
W20250112 20:51:48.531728 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a10f7a47089a43d8ae9c2fc16d379740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10f7a47089a43d8ae9c2fc16d379740 already in progress: creating tablet
W20250112 20:51:48.532356 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2aef567a7cf64178b53625ac4638048b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aef567a7cf64178b53625ac4638048b already in progress: creating tablet
W20250112 20:51:48.532588 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a2dbdbf416f64011b431f5188e0a06c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbdbf416f64011b431f5188e0a06c4 already in progress: creating tablet
I20250112 20:51:48.533427 1185 ts_tablet_manager.cc:1918] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:48.534220 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 53518b453d224ca186fcc6b2ffab5f04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53518b453d224ca186fcc6b2ffab5f04 already in progress: creating tablet
W20250112 20:51:48.534478 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 734734ff9c564f769bea0984a9e0386b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734734ff9c564f769bea0984a9e0386b already in progress: creating tablet
W20250112 20:51:48.534725 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet e92d58058511437d830a8f3cbba1e206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e92d58058511437d830a8f3cbba1e206 already in progress: creating tablet
W20250112 20:51:48.534934 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c6bdfecd5d9142d5b008c613864c8217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bdfecd5d9142d5b008c613864c8217 already in progress: creating tablet
W20250112 20:51:48.535176 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f0fb6a5146c44edea076a10da58ea1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fb6a5146c44edea076a10da58ea1a7 already in progress: creating tablet
W20250112 20:51:48.535389 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f424288fe62849edbbd2d38e89331fb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424288fe62849edbbd2d38e89331fb7 already in progress: creating tablet
W20250112 20:51:48.535609 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6158b97ccf0a42fdb1a510c5d1e80213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6158b97ccf0a42fdb1a510c5d1e80213 already in progress: creating tablet
I20250112 20:51:48.537511 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_93126de589c046d09407fb32545376cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.538029 1178 ts_tablet_manager.cc:1428] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
W20250112 20:51:48.538130 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93126de589c046d09407fb32545376cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.539572 983 leader_election.cc:304] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:48.540526 2964 raft_consensus.cc:2798] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.540658 2964 raft_consensus.cc:695] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.541139 2964 consensus_queue.cc:237] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.543371 1178 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:48.545159 1178 tablet_bootstrap.cc:654] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.550215 1178 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.550344 1178 ts_tablet_manager.cc:1397] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:48.559392 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98547168378947df885c601ccdc85e57" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:48.559808 983 leader_election.cc:343] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.559999 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.560235 913 raft_consensus.cc:2463] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:48.560731 983 leader_election.cc:343] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:48.561816 983 leader_election.cc:343] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.562546 983 leader_election.cc:304] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:48.562935 2787 raft_consensus.cc:3054] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.565387 2787 raft_consensus.cc:2743] T fa2f3389061143509354f37ae93ef0c7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:48.565896 983 leader_election.cc:343] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.561650 1164 tablet_bootstrap.cc:492] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.569934 1164 ts_tablet_manager.cc:1397] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.175s user 0.027s sys 0.035s
I20250112 20:51:48.570691 1164 raft_consensus.cc:357] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.570885 1164 raft_consensus.cc:383] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.571095 1164 raft_consensus.cc:738] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.571327 1164 consensus_queue.cc:260] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.571842 1164 ts_tablet_manager.cc:1428] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:48.572139 1164 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.574553 1164 tablet_bootstrap.cc:654] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.575930 1185 log.cc:1198] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:48.576372 1185 ts_tablet_manager.cc:1939] T 4d29316b9d2441bca97f56bdb0e56f0d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:48.576501 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5797c2fd8e21490ca00ef171965c31ad consensus state has no leader
I20250112 20:51:48.577167 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 913) took 3090 ms. Trace:
I20250112 20:51:48.578022 1185 rpcz_store.cc:276] 0112 20:51:45.487138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.487374 (+ 236us) service_pool.cc:225] Handling call
0112 20:51:48.161685 (+2674311us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:48.161697 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.475460 (+313763us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.533293 (+ 57833us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.577150 (+ 43857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2673035}
I20250112 20:51:48.578418 1185 tablet_replica.cc:331] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.578907 1185 raft_consensus.cc:2238] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.578981 1164 tablet_bootstrap.cc:492] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.579180 1164 ts_tablet_manager.cc:1397] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250112 20:51:48.579381 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 4d29316b9d2441bca97f56bdb0e56f0d (table default.Table_361fe8fc7b6845369a0ee99bc93d2771 [id=6278d7da530b4f14b2ad6de73057fdbf]) successfully deleted
I20250112 20:51:48.579505 1185 raft_consensus.cc:2267] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.579864 1164 raft_consensus.cc:357] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.580040 1164 raft_consensus.cc:383] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.580106 1164 raft_consensus.cc:738] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.580324 1164 consensus_queue.cc:260] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.580811 1164 ts_tablet_manager.cc:1428] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:48.581177 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c53abd31a1464c22abcb8d42a39cefac" } modify_external_catalogs: true
I20250112 20:51:48.581398 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c53abd31a1464c22abcb8d42a39cefac" } modify_external_catalogs: true
I20250112 20:51:48.581588 1164 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.582948 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.583216 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6763d75a41e34bc7aee13341771b4c7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.583724 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6763d75a41e34bc7aee13341771b4c7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.583886 1164 tablet_bootstrap.cc:654] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.585207 1185 ts_tablet_manager.cc:1905] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.588591 2521 raft_consensus.cc:2798] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.588675 2521 raft_consensus.cc:491] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.588742 2521 raft_consensus.cc:3054] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.591228 2521 raft_consensus.cc:513] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.591713 1185 ts_tablet_manager.cc:1918] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.591751 2521 leader_election.cc:290] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:48.591805 1185 log.cc:1198] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/01367a11dea04b33a9255508501b9060
I20250112 20:51:48.592622 1178 raft_consensus.cc:357] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.592798 1178 raft_consensus.cc:383] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.592842 1178 raft_consensus.cc:738] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.593007 1178 consensus_queue.cc:260] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.593464 1178 ts_tablet_manager.cc:1428] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250112 20:51:48.593722 1178 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:48.595052 1178 tablet_bootstrap.cc:654] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.595445 1185 ts_tablet_manager.cc:1939] T 01367a11dea04b33a9255508501b9060 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.596184 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 915) took 3090 ms. Trace:
I20250112 20:51:48.597287 1185 rpcz_store.cc:276] 0112 20:51:45.505568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.511493 (+ 5925us) service_pool.cc:225] Handling call
0112 20:51:48.578371 (+3066878us) ts_tablet_manager.cc:1145] Deleting tablet 01367a11dea04b33a9255508501b9060
0112 20:51:48.578379 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.588372 (+ 9993us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.591632 (+ 3260us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.596165 (+ 4533us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5149,"tablet-delete.queue_time_us":3066708}
I20250112 20:51:48.594048 1186 ts_tablet_manager.cc:1918] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.597529 1186 log.cc:1198] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b4ef01b41b6140d6bdad07bc79dd7dc5
I20250112 20:51:48.597927 1186 ts_tablet_manager.cc:1939] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.598227 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 918) took 3031 ms. Trace:
I20250112 20:51:48.598282 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 902) took 3535 ms. Trace:
I20250112 20:51:48.598371 1186 rpcz_store.cc:276] 0112 20:51:45.062845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.062950 (+ 105us) service_pool.cc:225] Handling call
0112 20:51:48.465281 (+3402331us) ts_tablet_manager.cc:1145] Deleting tablet b4ef01b41b6140d6bdad07bc79dd7dc5
0112 20:51:48.465289 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.557967 (+ 92678us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.593964 (+ 35997us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.598264 (+ 4300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3402199}
I20250112 20:51:48.598747 1186 tablet_replica.cc:331] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.598666 1185 rpcz_store.cc:276] 0112 20:51:45.566997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.577235 (+ 10238us) service_pool.cc:225] Handling call
0112 20:51:48.598093 (+3020858us) ts_tablet_manager.cc:1145] Deleting tablet fc1335522a384eb98163a2539630b93e
0112 20:51:48.598103 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.598209 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3020646}
I20250112 20:51:48.598935 1186 raft_consensus.cc:2238] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.599069 1186 raft_consensus.cc:2267] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.599318 1185 tablet_replica.cc:331] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.599542 1186 ts_tablet_manager.cc:1905] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.599761 1185 raft_consensus.cc:2238] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.600066 1185 raft_consensus.cc:2267] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.600911 1185 ts_tablet_manager.cc:1905] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:48.603149 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 79376.3us
I20250112 20:51:48.603608 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 01367a11dea04b33a9255508501b9060 (table default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]) successfully deleted
W20250112 20:51:48.605083 983 leader_election.cc:343] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:48.610455 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet fc1335522a384eb98163a2539630b93e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1335522a384eb98163a2539630b93e already in progress: creating tablet
I20250112 20:51:48.611616 1164 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.611742 1164 ts_tablet_manager.cc:1397] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.002s
I20250112 20:51:48.612371 1164 raft_consensus.cc:357] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.612555 1164 raft_consensus.cc:383] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.612613 1164 raft_consensus.cc:738] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.612880 1164 consensus_queue.cc:260] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.613612 1164 ts_tablet_manager.cc:1428] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:48.614403 1164 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.615895 1164 tablet_bootstrap.cc:654] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.617288 1185 ts_tablet_manager.cc:1918] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.617444 1185 log.cc:1198] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/cbc22207e9204b73877c74f61ebba8a8
I20250112 20:51:48.617832 1185 ts_tablet_manager.cc:1939] T cbc22207e9204b73877c74f61ebba8a8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.618546 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 923) took 2958 ms. Trace:
I20250112 20:51:48.618667 1185 rpcz_store.cc:276] 0112 20:51:45.659609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.660393 (+ 784us) service_pool.cc:225] Handling call
0112 20:51:48.599276 (+2938883us) ts_tablet_manager.cc:1145] Deleting tablet cbc22207e9204b73877c74f61ebba8a8
0112 20:51:48.599284 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.606697 (+ 7413us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.617198 (+ 10501us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.618528 (+ 1330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2938543}
I20250112 20:51:48.619191 1185 tablet_replica.cc:331] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.619421 1185 raft_consensus.cc:2238] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.619611 1185 raft_consensus.cc:2267] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.620311 1185 ts_tablet_manager.cc:1905] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.619277 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet dfc63eff44414906a027a8208b5cc5a9
I20250112 20:51:48.628717 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.628973 907 raft_consensus.cc:2463] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:48.629868 1164 tablet_bootstrap.cc:492] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.629994 1164 ts_tablet_manager.cc:1397] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20250112 20:51:48.630589 1164 raft_consensus.cc:357] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.630753 1164 raft_consensus.cc:383] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.630796 1164 raft_consensus.cc:738] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.630977 1164 consensus_queue.cc:260] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.631496 1164 ts_tablet_manager.cc:1428] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:48.631943 1164 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.633184 1164 tablet_bootstrap.cc:654] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.634989 1185 ts_tablet_manager.cc:1918] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:48.634994 847 leader_election.cc:343] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.635171 1185 log.cc:1198] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1d97ac93467a4a7a8ae44d52719f191c
I20250112 20:51:48.635572 1185 ts_tablet_manager.cc:1939] T 1d97ac93467a4a7a8ae44d52719f191c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.637696 1164 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.637866 1164 ts_tablet_manager.cc:1397] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250112 20:51:48.638357 2447 ts_tablet_manager.cc:1918] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.638460 2447 log.cc:1198] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8229e4ef771b4032a4bfae3190a851e3
I20250112 20:51:48.638779 2447 ts_tablet_manager.cc:1939] T 8229e4ef771b4032a4bfae3190a851e3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:48.642004 985 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.60:45153, user_credentials={real_user=slave}} blocked reactor thread for 57800.7us
W20250112 20:51:48.643720 847 leader_election.cc:343] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.645293 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 924) took 2985 ms. Trace:
I20250112 20:51:48.645376 1185 rpcz_store.cc:276] 0112 20:51:45.659712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.660587 (+ 875us) service_pool.cc:225] Handling call
0112 20:51:48.619148 (+2958561us) ts_tablet_manager.cc:1145] Deleting tablet 1d97ac93467a4a7a8ae44d52719f191c
0112 20:51:48.619157 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.624365 (+ 5208us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.634905 (+ 10540us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.645275 (+ 10370us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8850,"tablet-delete.queue_time_us":2958299}
I20250112 20:51:48.645759 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 930) took 2718 ms. Trace:
I20250112 20:51:48.645828 1185 rpcz_store.cc:276] 0112 20:51:45.927385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.927455 (+ 70us) service_pool.cc:225] Handling call
0112 20:51:48.645637 (+2718182us) ts_tablet_manager.cc:1145] Deleting tablet be1f12369c894682b7d30de4e37bd466
0112 20:51:48.645646 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.645742 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2717990}
I20250112 20:51:48.646152 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 931) took 2717 ms. Trace:
I20250112 20:51:48.646215 1185 rpcz_store.cc:276] 0112 20:51:45.928759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.928826 (+ 67us) service_pool.cc:225] Handling call
0112 20:51:48.646053 (+2717227us) ts_tablet_manager.cc:1145] Deleting tablet a10f7a47089a43d8ae9c2fc16d379740
0112 20:51:48.646061 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.646137 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2717062}
I20250112 20:51:48.646512 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 932) took 2708 ms. Trace:
I20250112 20:51:48.646994 2923 raft_consensus.cc:491] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.647115 2923 raft_consensus.cc:513] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.647280 1164 raft_consensus.cc:357] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.647439 1164 raft_consensus.cc:383] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.647480 1164 raft_consensus.cc:738] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.647608 2923 leader_election.cc:290] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.647723 2923 raft_consensus.cc:491] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.647802 2923 raft_consensus.cc:513] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.646574 1185 rpcz_store.cc:276] 0112 20:51:45.937891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.938525 (+ 634us) service_pool.cc:225] Handling call
0112 20:51:48.646395 (+2707870us) ts_tablet_manager.cc:1145] Deleting tablet 2aef567a7cf64178b53625ac4638048b
0112 20:51:48.646403 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.646496 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707714}
I20250112 20:51:48.648248 2923 leader_election.cc:290] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.648273 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 934) took 2622 ms. Trace:
I20250112 20:51:48.648337 1185 rpcz_store.cc:276] 0112 20:51:46.026106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.026381 (+ 275us) service_pool.cc:225] Handling call
0112 20:51:48.648162 (+2621781us) ts_tablet_manager.cc:1145] Deleting tablet a2dbdbf416f64011b431f5188e0a06c4
0112 20:51:48.648170 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.648260 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621127}
I20250112 20:51:48.648584 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 935) took 2616 ms. Trace:
I20250112 20:51:48.647629 1164 consensus_queue.cc:260] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.648645 1185 rpcz_store.cc:276] 0112 20:51:46.032621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.032664 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:48.648503 (+2615839us) ts_tablet_manager.cc:1145] Deleting tablet 53518b453d224ca186fcc6b2ffab5f04
0112 20:51:48.648510 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.648573 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2615679}
I20250112 20:51:48.648907 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 937) took 2606 ms. Trace:
I20250112 20:51:48.648962 1185 rpcz_store.cc:276] 0112 20:51:46.042201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.042261 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:48.648822 (+2606561us) ts_tablet_manager.cc:1145] Deleting tablet 734734ff9c564f769bea0984a9e0386b
0112 20:51:48.648829 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.648894 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2606396}
I20250112 20:51:48.649089 1164 ts_tablet_manager.cc:1428] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250112 20:51:48.649181 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 938) took 2517 ms. Trace:
I20250112 20:51:48.649334 1164 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.650905 1164 tablet_bootstrap.cc:654] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.649350 1185 rpcz_store.cc:276] 0112 20:51:46.131419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.131481 (+ 62us) service_pool.cc:225] Handling call
0112 20:51:48.649114 (+2517633us) ts_tablet_manager.cc:1145] Deleting tablet e92d58058511437d830a8f3cbba1e206
0112 20:51:48.649120 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.649172 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2517484}
I20250112 20:51:48.651885 1185 tablet_replica.cc:331] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.652101 1185 raft_consensus.cc:2238] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.652323 1185 pending_rounds.cc:62] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:48.654781 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b4a1d95f60f74be2a1f424c514f6efc3" } new_table_name: "default.Table_fd620909c9f04ca78577d1154703e893" modify_external_catalogs: true
I20250112 20:51:48.655124 1185 pending_rounds.cc:69] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585083872272384 op_type: NO_OP noop_request { }
I20250112 20:51:48.655227 1185 raft_consensus.cc:2883] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:48.655293 1185 raft_consensus.cc:2267] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.656239 1185 ts_tablet_manager.cc:1905] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.656227 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_04069d4b7bad4757988c3e7e8ab02b44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.656750 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04069d4b7bad4757988c3e7e8ab02b44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.658828 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_662121681f354c6a9f08a434efbafcce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.659303 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662121681f354c6a9f08a434efbafcce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.660372 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7e596a69a2ad4a0abbc10f0b7f9686fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.660809 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e596a69a2ad4a0abbc10f0b7f9686fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.662264 1185 ts_tablet_manager.cc:1918] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.662382 1185 log.cc:1198] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1fd47d1c1a4b42bdbff97f619dc95134
I20250112 20:51:48.662768 1185 ts_tablet_manager.cc:1939] T 1fd47d1c1a4b42bdbff97f619dc95134 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:48.663197 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 939) took 2249 ms. Trace:
I20250112 20:51:48.663277 1185 rpcz_store.cc:276] 0112 20:51:46.413697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.413746 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:48.651769 (+2238023us) ts_tablet_manager.cc:1145] Deleting tablet 1fd47d1c1a4b42bdbff97f619dc95134
0112 20:51:48.651777 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.659194 (+ 7417us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.662182 (+ 2988us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.663179 (+ 997us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21,"tablet-delete.queue_time_us":2237864}
I20250112 20:51:48.664255 1186 ts_tablet_manager.cc:1918] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.664230 744 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc63eff44414906a027a8208b5cc5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.664356 1186 log.cc:1198] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1fd34e829d0645ea80daaff5d27cb266
I20250112 20:51:48.664601 1186 ts_tablet_manager.cc:1939] T 1fd34e829d0645ea80daaff5d27cb266 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.664891 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet cbc22207e9204b73877c74f61ebba8a8 (table default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]) successfully deleted
I20250112 20:51:48.665196 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1d97ac93467a4a7a8ae44d52719f191c (table default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]) successfully deleted
W20250112 20:51:48.665409 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet be1f12369c894682b7d30de4e37bd466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1f12369c894682b7d30de4e37bd466 already in progress: creating tablet
W20250112 20:51:48.665611 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a10f7a47089a43d8ae9c2fc16d379740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10f7a47089a43d8ae9c2fc16d379740 already in progress: creating tablet
W20250112 20:51:48.665817 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2aef567a7cf64178b53625ac4638048b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aef567a7cf64178b53625ac4638048b already in progress: creating tablet
W20250112 20:51:48.666018 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a2dbdbf416f64011b431f5188e0a06c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbdbf416f64011b431f5188e0a06c4 already in progress: creating tablet
W20250112 20:51:48.666249 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 53518b453d224ca186fcc6b2ffab5f04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53518b453d224ca186fcc6b2ffab5f04 already in progress: creating tablet
W20250112 20:51:48.666452 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 734734ff9c564f769bea0984a9e0386b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734734ff9c564f769bea0984a9e0386b already in progress: creating tablet
W20250112 20:51:48.666661 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet e92d58058511437d830a8f3cbba1e206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e92d58058511437d830a8f3cbba1e206 already in progress: creating tablet
I20250112 20:51:48.666864 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1fd47d1c1a4b42bdbff97f619dc95134 (table default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]) successfully deleted
I20250112 20:51:48.667603 2923 raft_consensus.cc:491] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.667726 2923 raft_consensus.cc:513] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.668298 2923 leader_election.cc:290] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.668547 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 904) took 3605 ms. Trace:
I20250112 20:51:48.668637 1186 rpcz_store.cc:276] 0112 20:51:45.063115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.063150 (+ 35us) service_pool.cc:225] Handling call
0112 20:51:48.598695 (+3535545us) ts_tablet_manager.cc:1145] Deleting tablet 1fd34e829d0645ea80daaff5d27cb266
0112 20:51:48.598707 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.645471 (+ 46764us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.664169 (+ 18698us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.668529 (+ 4360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3535412}
I20250112 20:51:48.668926 1186 tablet_replica.cc:331] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.669086 1186 raft_consensus.cc:2238] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.669310 710 leader_election.cc:304] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:48.669533 2923 raft_consensus.cc:2743] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:48.670720 710 leader_election.cc:304] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:48.670953 2923 raft_consensus.cc:2743] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:48.671384 710 leader_election.cc:304] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:48.671598 2886 raft_consensus.cc:2743] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 20:51:48.673864 1178 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.674008 1178 ts_tablet_manager.cc:1397] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.005s
I20250112 20:51:48.674592 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.674810 1042 raft_consensus.cc:2463] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:48.676276 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc63eff44414906a027a8208b5cc5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.677029 1186 raft_consensus.cc:2267] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.677520 1186 ts_tablet_manager.cc:1905] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.682546 1178 raft_consensus.cc:357] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.682718 1178 raft_consensus.cc:383] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.682761 1178 raft_consensus.cc:738] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.682961 1178 consensus_queue.cc:260] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.684445 1186 ts_tablet_manager.cc:1918] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.684536 1186 log.cc:1198] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1fd47d1c1a4b42bdbff97f619dc95134
I20250112 20:51:48.684837 1186 ts_tablet_manager.cc:1939] T 1fd47d1c1a4b42bdbff97f619dc95134 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.685117 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 903) took 3622 ms. Trace:
I20250112 20:51:48.685192 1186 rpcz_store.cc:276] 0112 20:51:45.062971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.063494 (+ 523us) service_pool.cc:225] Handling call
0112 20:51:48.668883 (+3605389us) ts_tablet_manager.cc:1145] Deleting tablet 1fd47d1c1a4b42bdbff97f619dc95134
0112 20:51:48.668892 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.680538 (+ 11646us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.684371 (+ 3833us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.685104 (+ 733us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7758,"tablet-delete.queue_time_us":3605242}
I20250112 20:51:48.685537 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 905) took 3602 ms. Trace:
I20250112 20:51:48.685602 1186 rpcz_store.cc:276] 0112 20:51:45.082690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.086269 (+ 3579us) service_pool.cc:225] Handling call
0112 20:51:48.685433 (+3599164us) ts_tablet_manager.cc:1145] Deleting tablet e8bea4d83ea644a0930a29f61e7a5fc8
0112 20:51:48.685442 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.685524 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3598952}
I20250112 20:51:48.685809 1186 tablet_replica.cc:331] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.685973 1186 raft_consensus.cc:2238] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.687043 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b4de1d4b39114ec2b50c35a8a02318f9" } new_table_name: "default.Table_4702b97eedb5450bb0cff7a0108b5519" modify_external_catalogs: true
I20250112 20:51:48.688227 1178 ts_tablet_manager.cc:1428] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 20:51:48.689496 1178 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
W20250112 20:51:48.689981 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f75f9480f6684d1cb710a30844e06e0e consensus state has no leader
I20250112 20:51:48.690747 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ed925a5ef3a54f819ca292a4e7967b33
I20250112 20:51:48.690927 1178 tablet_bootstrap.cc:654] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.691082 2983 consensus_queue.cc:1035] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.692265 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.692562 1186 raft_consensus.cc:2267] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.693053 1186 ts_tablet_manager.cc:1905] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.700536 1178 tablet_bootstrap.cc:492] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.700655 1178 ts_tablet_manager.cc:1397] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250112 20:51:48.701270 1178 raft_consensus.cc:357] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.701426 1178 raft_consensus.cc:383] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.701469 1178 raft_consensus.cc:738] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.703060 1178 consensus_queue.cc:260] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.704135 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c57b973f9ff045d687f8b5609c1941fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.704653 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c57b973f9ff045d687f8b5609c1941fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.706185 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eb174dc18b7d43d69147c2d69b5f458d" } new_table_name: "default.Table_55404e958b16400f9a4f336dc5f3cd32" modify_external_catalogs: true
I20250112 20:51:48.708019 1178 ts_tablet_manager.cc:1428] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250112 20:51:48.708731 1178 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:48.709993 1164 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.710145 1164 ts_tablet_manager.cc:1397] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.002s
I20250112 20:51:48.710808 1164 raft_consensus.cc:357] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.710991 1164 raft_consensus.cc:383] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.711053 1164 raft_consensus.cc:738] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.711241 1164 consensus_queue.cc:260] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.711433 1186 ts_tablet_manager.cc:1918] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.711516 1186 log.cc:1198] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:48.711699 1164 ts_tablet_manager.cc:1428] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 20:51:48.711766 1186 ts_tablet_manager.cc:1939] T 8c5b10576bd34af5b02f8d8dcc3759ab P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.712714 1164 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.714067 1164 tablet_bootstrap.cc:654] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.716323 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 906) took 3505 ms. Trace:
I20250112 20:51:48.716418 1186 rpcz_store.cc:276] 0112 20:51:45.210558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.210632 (+ 74us) service_pool.cc:225] Handling call
0112 20:51:48.685776 (+3475144us) ts_tablet_manager.cc:1145] Deleting tablet 8c5b10576bd34af5b02f8d8dcc3759ab
0112 20:51:48.685783 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.695875 (+ 10092us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.711349 (+ 15474us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.716303 (+ 4954us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6455,"tablet-delete.queue_time_us":3474936}
I20250112 20:51:48.717041 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6f7b30f2842340de9879b5e275a9c8a4" } modify_external_catalogs: true
I20250112 20:51:48.717275 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6f7b30f2842340de9879b5e275a9c8a4" } modify_external_catalogs: true
I20250112 20:51:48.718483 1178 tablet_bootstrap.cc:654] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.720953 1164 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.721237 1164 ts_tablet_manager.cc:1397] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250112 20:51:48.721853 1164 raft_consensus.cc:357] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.722031 1164 raft_consensus.cc:383] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.722141 1164 raft_consensus.cc:738] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.722350 1164 consensus_queue.cc:260] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.723511 1164 ts_tablet_manager.cc:1428] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 20:51:48.723794 983 consensus_peers.cc:487] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b8cc99742f4f9ba68eb1aa17cab248. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:48.724187 1164 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.725687 1164 tablet_bootstrap.cc:654] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.727396 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_db5f5dc69d334043b179fca5b701f975" } modify_external_catalogs: true
I20250112 20:51:48.727618 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_db5f5dc69d334043b179fca5b701f975" } modify_external_catalogs: true
I20250112 20:51:48.729625 1186 tablet_replica.cc:331] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.729825 1186 raft_consensus.cc:2238] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.729998 1186 raft_consensus.cc:2267] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.730552 1186 ts_tablet_manager.cc:1905] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.734573 2757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:48.736631 1186 ts_tablet_manager.cc:1918] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.736737 1186 log.cc:1198] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/89f9b46da06a476ab9f2a93ca74ff736
I20250112 20:51:48.737118 1186 ts_tablet_manager.cc:1939] T 89f9b46da06a476ab9f2a93ca74ff736 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.737416 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 908) took 3325 ms. Trace:
I20250112 20:51:48.737500 1186 rpcz_store.cc:276] 0112 20:51:45.411764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.411819 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:48.716709 (+3304890us) ts_tablet_manager.cc:1145] Deleting tablet 89f9b46da06a476ab9f2a93ca74ff736
0112 20:51:48.729579 (+ 12870us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.733791 (+ 4212us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.736551 (+ 2760us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.737401 (+ 850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3304709}
I20250112 20:51:48.738718 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 909) took 3314 ms. Trace:
I20250112 20:51:48.738797 1186 rpcz_store.cc:276] 0112 20:51:45.424680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.424849 (+ 169us) service_pool.cc:225] Handling call
0112 20:51:48.738248 (+3313399us) ts_tablet_manager.cc:1145] Deleting tablet 015f54bdae6b4333963f9e085c5189e8
0112 20:51:48.738259 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.738701 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313228}
I20250112 20:51:48.739115 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 910) took 3306 ms. Trace:
I20250112 20:51:48.739114 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9989f3dc61df4f75b9bdb85ba3ca3f37" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.739192 1186 rpcz_store.cc:276] 0112 20:51:45.432815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.432860 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:48.739010 (+3306150us) ts_tablet_manager.cc:1145] Deleting tablet 0d072ebae78e40c48e158eb65775476c
0112 20:51:48.739019 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.739091 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3305938}
I20250112 20:51:48.739367 1042 raft_consensus.cc:3054] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.739478 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 911) took 3298 ms. Trace:
I20250112 20:51:48.739539 1186 rpcz_store.cc:276] 0112 20:51:45.441182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.441223 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:48.739391 (+3298168us) ts_tablet_manager.cc:1145] Deleting tablet 3f6ad2ac50c745448031bdf94e0eea59
0112 20:51:48.739398 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.739465 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3298014}
I20250112 20:51:48.739799 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 912) took 3288 ms. Trace:
I20250112 20:51:48.739864 1186 rpcz_store.cc:276] 0112 20:51:45.451150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.452197 (+ 1047us) service_pool.cc:225] Handling call
0112 20:51:48.739710 (+3287513us) ts_tablet_manager.cc:1145] Deleting tablet 69012f97aac1441095aa3bb1548de9cd
0112 20:51:48.739718 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.739785 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3287366}
I20250112 20:51:48.740079 1186 tablet_replica.cc:331] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.740253 1186 raft_consensus.cc:2238] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.740389 1186 raft_consensus.cc:2267] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.740927 1186 ts_tablet_manager.cc:1905] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.743299 1042 raft_consensus.cc:2463] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:48.744477 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6302bcdd216e470db6dbad8bbcfee6f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.744673 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 907) took 2034 ms. Trace:
I20250112 20:51:48.744771 2447 rpcz_store.cc:276] 0112 20:51:46.709879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.709927 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:48.478178 (+1768251us) ts_tablet_manager.cc:1145] Deleting tablet 8229e4ef771b4032a4bfae3190a851e3
0112 20:51:48.478187 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.517912 (+ 39725us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.638272 (+120360us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.744649 (+106377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768105}
W20250112 20:51:48.745245 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6302bcdd216e470db6dbad8bbcfee6f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.745306 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 908) took 2034 ms. Trace:
I20250112 20:51:48.745385 2447 rpcz_store.cc:276] 0112 20:51:46.710732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.710781 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:48.745172 (+2034391us) ts_tablet_manager.cc:1145] Deleting tablet 2de03e3114d145aeb81590dbae2c548e
0112 20:51:48.745185 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.745289 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034253}
I20250112 20:51:48.745676 2447 tablet_replica.cc:331] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:48.745927 2447 raft_consensus.cc:2238] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.746078 2447 raft_consensus.cc:2267] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.746341 1186 ts_tablet_manager.cc:1918] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.746438 1186 log.cc:1198] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/c2664a2e98f74f73ab2dc0adaa2335d3
I20250112 20:51:48.746652 2447 ts_tablet_manager.cc:1905] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.746739 1186 ts_tablet_manager.cc:1939] T c2664a2e98f74f73ab2dc0adaa2335d3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.747028 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 914) took 3271 ms. Trace:
I20250112 20:51:48.747099 1186 rpcz_store.cc:276] 0112 20:51:45.475947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.477105 (+ 1158us) service_pool.cc:225] Handling call
0112 20:51:48.740044 (+3262939us) ts_tablet_manager.cc:1145] Deleting tablet c2664a2e98f74f73ab2dc0adaa2335d3
0112 20:51:48.740051 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.743595 (+ 3544us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.746256 (+ 2661us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.747014 (+ 758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3262754}
I20250112 20:51:48.747424 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 915) took 3261 ms. Trace:
I20250112 20:51:48.747499 1186 rpcz_store.cc:276] 0112 20:51:45.485535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.493507 (+ 7972us) service_pool.cc:225] Handling call
0112 20:51:48.747331 (+3253824us) ts_tablet_manager.cc:1145] Deleting tablet 3b0786594542416cb67c4112921943b3
0112 20:51:48.747339 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.747409 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3244475}
I20250112 20:51:48.747730 1186 tablet_replica.cc:331] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.747936 1186 raft_consensus.cc:2238] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.748095 1186 raft_consensus.cc:2267] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.748587 1186 ts_tablet_manager.cc:1905] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.750403 744 tablet_service.cc:1514] Processing DeleteTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.751785 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8229e4ef771b4032a4bfae3190a851e3 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
W20250112 20:51:48.752136 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2de03e3114d145aeb81590dbae2c548e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de03e3114d145aeb81590dbae2c548e already in progress: creating tablet
I20250112 20:51:48.752657 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet cbc22207e9204b73877c74f61ebba8a8 (table default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]) successfully deleted
I20250112 20:51:48.753917 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.755146 710 leader_election.cc:304] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:48.755420 2886 raft_consensus.cc:2743] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:48.762658 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 940) took 2317 ms. Trace:
I20250112 20:51:48.762859 1185 rpcz_store.cc:276] 0112 20:51:46.445178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.445230 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:48.762489 (+2317259us) ts_tablet_manager.cc:1145] Deleting tablet c6bdfecd5d9142d5b008c613864c8217
0112 20:51:48.762506 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.762633 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2317018}
I20250112 20:51:48.763540 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 941) took 2269 ms. Trace:
I20250112 20:51:48.763697 1185 rpcz_store.cc:276] 0112 20:51:46.494273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.494327 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:48.763397 (+2269070us) ts_tablet_manager.cc:1145] Deleting tablet f0fb6a5146c44edea076a10da58ea1a7
0112 20:51:48.763412 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.763520 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2268909}
I20250112 20:51:48.764292 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 942) took 2267 ms. Trace:
I20250112 20:51:48.764437 1185 rpcz_store.cc:276] 0112 20:51:46.496503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.496763 (+ 260us) service_pool.cc:225] Handling call
0112 20:51:48.764179 (+2267416us) ts_tablet_manager.cc:1145] Deleting tablet f424288fe62849edbbd2d38e89331fb7
0112 20:51:48.764190 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.764275 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267182}
I20250112 20:51:48.765055 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 952) took 2056 ms. Trace:
I20250112 20:51:48.765203 1185 rpcz_store.cc:276] 0112 20:51:46.708717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.708956 (+ 239us) service_pool.cc:225] Handling call
0112 20:51:48.764937 (+2055981us) ts_tablet_manager.cc:1145] Deleting tablet 6158b97ccf0a42fdb1a510c5d1e80213
0112 20:51:48.764949 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.765038 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055772}
I20250112 20:51:48.765709 1185 tablet_replica.cc:331] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:48.766012 1185 raft_consensus.cc:2238] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.766409 1185 raft_consensus.cc:2267] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.767174 1185 ts_tablet_manager.cc:1905] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:48.775107 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd62b97f353e437dab548c679b28e446 consensus state has no leader
I20250112 20:51:48.775442 454 master_service.cc:946] ReplaceTablet: received request to replace tablet b4ef01b41b6140d6bdad07bc79dd7dc5 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:48.777750 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_604c0839efdd41cc8400741910f59256" } new_table_name: "default.Table_cb2b5abd46bc4877887604a94159e418" modify_external_catalogs: true
I20250112 20:51:48.789116 2787 consensus_queue.cc:1035] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.790004 1186 ts_tablet_manager.cc:1918] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.790128 1186 log.cc:1198] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/01367a11dea04b33a9255508501b9060
I20250112 20:51:48.790437 1186 ts_tablet_manager.cc:1939] T 01367a11dea04b33a9255508501b9060 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.790735 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 917) took 3284 ms. Trace:
I20250112 20:51:48.790796 1186 rpcz_store.cc:276] 0112 20:51:45.506031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.511703 (+ 5672us) service_pool.cc:225] Handling call
0112 20:51:48.747694 (+3235991us) ts_tablet_manager.cc:1145] Deleting tablet 01367a11dea04b33a9255508501b9060
0112 20:51:48.747702 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.786932 (+ 39230us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.789906 (+ 2974us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.790722 (+ 816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3235838}
I20250112 20:51:48.791183 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 920) took 3224 ms. Trace:
I20250112 20:51:48.791255 1186 rpcz_store.cc:276] 0112 20:51:45.567005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.567065 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:48.791037 (+3223972us) ts_tablet_manager.cc:1145] Deleting tablet fc1335522a384eb98163a2539630b93e
0112 20:51:48.791064 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.791169 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3223801}
I20250112 20:51:48.791530 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 930) took 2865 ms. Trace:
I20250112 20:51:48.791604 1186 rpcz_store.cc:276] 0112 20:51:45.926556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.926763 (+ 207us) service_pool.cc:225] Handling call
0112 20:51:48.791434 (+2864671us) ts_tablet_manager.cc:1145] Deleting tablet be1f12369c894682b7d30de4e37bd466
0112 20:51:48.791443 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.791517 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2863709}
I20250112 20:51:48.791889 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 931) took 2862 ms. Trace:
I20250112 20:51:48.791987 1186 rpcz_store.cc:276] 0112 20:51:45.929480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.929528 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:48.791804 (+2862276us) ts_tablet_manager.cc:1145] Deleting tablet a10f7a47089a43d8ae9c2fc16d379740
0112 20:51:48.791812 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.791875 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862116}
I20250112 20:51:48.792290 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 932) took 2862 ms. Trace:
I20250112 20:51:48.792366 1186 rpcz_store.cc:276] 0112 20:51:45.930169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:45.930207 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:48.792202 (+2861995us) ts_tablet_manager.cc:1145] Deleting tablet 2aef567a7cf64178b53625ac4638048b
0112 20:51:48.792210 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.792277 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2861861}
I20250112 20:51:48.792627 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 933) took 2766 ms. Trace:
I20250112 20:51:48.792690 1186 rpcz_store.cc:276] 0112 20:51:46.026304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.026399 (+ 95us) service_pool.cc:225] Handling call
0112 20:51:48.792544 (+2766145us) ts_tablet_manager.cc:1145] Deleting tablet a2dbdbf416f64011b431f5188e0a06c4
0112 20:51:48.792552 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.792613 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2765918}
I20250112 20:51:48.797678 2787 raft_consensus.cc:491] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.797839 2787 raft_consensus.cc:513] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.798581 2787 leader_election.cc:290] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.798768 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 934) took 2764 ms. Trace:
I20250112 20:51:48.798866 1186 rpcz_store.cc:276] 0112 20:51:46.034003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.034076 (+ 73us) service_pool.cc:225] Handling call
0112 20:51:48.798637 (+2764561us) ts_tablet_manager.cc:1145] Deleting tablet 53518b453d224ca186fcc6b2ffab5f04
0112 20:51:48.798651 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.798748 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758644}
I20250112 20:51:48.799242 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 935) took 2757 ms. Trace:
I20250112 20:51:48.799573 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4a237cfeeb4d0fb7ad1abb4ff97845" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.799808 913 raft_consensus.cc:2463] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.799320 1186 rpcz_store.cc:276] 0112 20:51:46.042173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.042238 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:48.799134 (+2756896us) ts_tablet_manager.cc:1145] Deleting tablet 734734ff9c564f769bea0984a9e0386b
0112 20:51:48.799144 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.799225 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2756714}
I20250112 20:51:48.800232 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 936) took 2672 ms. Trace:
I20250112 20:51:48.800315 1186 rpcz_store.cc:276] 0112 20:51:46.127761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.127840 (+ 79us) service_pool.cc:225] Handling call
0112 20:51:48.800127 (+2672287us) ts_tablet_manager.cc:1145] Deleting tablet e92d58058511437d830a8f3cbba1e206
0112 20:51:48.800138 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.800215 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672112}
I20250112 20:51:48.800406 983 leader_election.cc:304] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.800581 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 938) took 2346 ms. Trace:
I20250112 20:51:48.800662 1186 rpcz_store.cc:276] 0112 20:51:46.453779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.453845 (+ 66us) service_pool.cc:225] Handling call
0112 20:51:48.800500 (+2346655us) ts_tablet_manager.cc:1145] Deleting tablet c6bdfecd5d9142d5b008c613864c8217
0112 20:51:48.800507 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.800566 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346471}
I20250112 20:51:48.800841 2787 raft_consensus.cc:2798] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.800966 2787 raft_consensus.cc:491] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.800989 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 939) took 2345 ms. Trace:
I20250112 20:51:48.801060 2787 raft_consensus.cc:3054] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.801071 1186 rpcz_store.cc:276] 0112 20:51:46.455382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.455437 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:48.800889 (+2345452us) ts_tablet_manager.cc:1145] Deleting tablet f0fb6a5146c44edea076a10da58ea1a7
0112 20:51:48.800897 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.800974 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2345258}
I20250112 20:51:48.801409 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 940) took 2286 ms. Trace:
I20250112 20:51:48.801477 1186 rpcz_store.cc:276] 0112 20:51:46.514916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.514971 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:48.801312 (+2286341us) ts_tablet_manager.cc:1145] Deleting tablet f424288fe62849edbbd2d38e89331fb7
0112 20:51:48.801322 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.801394 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283654}
I20250112 20:51:48.801828 1186 tablet_replica.cc:331] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.802043 1186 raft_consensus.cc:2238] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:48.802289 1186 pending_rounds.cc:62] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:48.802337 1186 pending_rounds.cc:69] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585082045575168 op_type: NO_OP noop_request { }
I20250112 20:51:48.802429 1186 raft_consensus.cc:2883] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:48.802497 1186 raft_consensus.cc:2267] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.804118 1186 ts_tablet_manager.cc:1905] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.804306 2787 raft_consensus.cc:513] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.805183 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:48.805522 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d477fb5e5ad5447bb182bc9998561a43 (table default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49]). Expected version 2 got 0
I20250112 20:51:48.805712 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]). Expected version 1 got 0
I20250112 20:51:48.806061 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]). Expected version 2 got 0
I20250112 20:51:48.806099 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4a237cfeeb4d0fb7ad1abb4ff97845" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.806303 913 raft_consensus.cc:3054] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.806655 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:48.807036 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:48.807184 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1e584b761d234c5aa0ab875acd8e1e05 (table default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a]). Expected version 4 got 0
I20250112 20:51:48.807437 452 catalog_manager.cc:5526] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.808292 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:48.808635 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:48.808893 2787 leader_election.cc:290] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.808915 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623] alter complete (version 1)
I20250112 20:51:48.810026 913 raft_consensus.cc:2463] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.810561 983 leader_election.cc:304] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:48.811229 1186 ts_tablet_manager.cc:1918] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:48.811355 1186 log.cc:1198] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/c8d24bccf682483aa9b273fc1ce847da
I20250112 20:51:48.811446 2787 raft_consensus.cc:2798] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.811709 1186 ts_tablet_manager.cc:1939] T c8d24bccf682483aa9b273fc1ce847da P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.811980 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 942) took 2295 ms. Trace:
I20250112 20:51:48.812044 1186 rpcz_store.cc:276] 0112 20:51:46.516207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.516265 (+ 58us) service_pool.cc:225] Handling call
0112 20:51:48.801779 (+2285514us) ts_tablet_manager.cc:1145] Deleting tablet c8d24bccf682483aa9b273fc1ce847da
0112 20:51:48.801790 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.807936 (+ 6146us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.811073 (+ 3137us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.811969 (+ 896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283999}
I20250112 20:51:48.812985 2787 raft_consensus.cc:695] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:48.813272 2787 consensus_queue.cc:237] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.813683 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3846863c74f64923bda6e5a19e7f2770
I20250112 20:51:48.812310 1186 tablet_replica.cc:331] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.820665 1186 raft_consensus.cc:2238] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.820843 1186 raft_consensus.cc:2267] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.821393 1186 ts_tablet_manager.cc:1905] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.832096 1186 ts_tablet_manager.cc:1918] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.832264 1186 log.cc:1198] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/669e1cfd407e4d87a63e6832d9c6fa1f
I20250112 20:51:48.832571 1186 ts_tablet_manager.cc:1939] T 669e1cfd407e4d87a63e6832d9c6fa1f P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.834187 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 3846863c74f64923bda6e5a19e7f2770 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.838771 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 941) took 2322 ms. Trace:
I20250112 20:51:48.838863 1186 rpcz_store.cc:276] 0112 20:51:46.515981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.516657 (+ 676us) service_pool.cc:225] Handling call
0112 20:51:48.812273 (+2295616us) ts_tablet_manager.cc:1145] Deleting tablet 669e1cfd407e4d87a63e6832d9c6fa1f
0112 20:51:48.812280 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.824266 (+ 11986us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.832001 (+ 7735us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.838753 (+ 6752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294366}
I20250112 20:51:48.839202 1186 tablet_replica.cc:331] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.839386 1186 raft_consensus.cc:2238] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.839537 1186 raft_consensus.cc:2267] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.840075 1186 ts_tablet_manager.cc:1905] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.843667 2757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:48.847024 2787 consensus_queue.cc:1035] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.848711 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet c2664a2e98f74f73ab2dc0adaa2335d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.848850 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.848970 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.849210 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.849403 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:48.852113 1178 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:48.852283 1178 ts_tablet_manager.cc:1397] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.144s user 0.036s sys 0.079s
I20250112 20:51:48.852985 1178 raft_consensus.cc:357] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.853209 1178 raft_consensus.cc:383] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.853293 1178 raft_consensus.cc:738] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.853511 1178 consensus_queue.cc:260] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.854135 1178 ts_tablet_manager.cc:1428] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:48.854399 1178 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:48.855777 1178 tablet_bootstrap.cc:654] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.865626 2757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:48.878351 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 (DEFAULT_TABLE table=default.Table_137d3f9986ea49519c9585b3761b586c [id=e697ab0e40864bd49fbc206de2bbf227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.878798 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7959f3ed319f4078ab747d490dce7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.877434 1186 ts_tablet_manager.cc:1918] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:48.881122 1186 log.cc:1198] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/7f807071e53a4f0e8629b1dfa6d26b9d
I20250112 20:51:48.881438 1186 ts_tablet_manager.cc:1939] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:48.883458 2787 consensus_queue.cc:1035] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:48.883736 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 720aa779aada46d5a888d5011804a6a8 consensus state has no leader
W20250112 20:51:48.816900 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 61251.4us
I20250112 20:51:48.833616 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3846863c74f64923bda6e5a19e7f2770 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:48.853835 2886 raft_consensus.cc:491] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.887604 2886 raft_consensus.cc:513] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.888293 2886 leader_election.cc:290] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:48.865878 1164 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:48.877763 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 (DEFAULT_TABLE table=default.Table_137d3f9986ea49519c9585b3761b586c [id=e697ab0e40864bd49fbc206de2bbf227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.891359 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7959f3ed319f4078ab747d490dce7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.885797 753 tablet_service.cc:1467] Processing CreateTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 (DEFAULT_TABLE table=default.Table_4a7a33e2dd7b42ada35718df1413bdd4 [id=70aa77642969417ca31d9b112034370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.892848 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0866871b9e1542e8ab30505c68f7dea9
I20250112 20:51:48.893137 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096ee7d3a66846f9b619d8f2fc046616. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.897109 1164 ts_tablet_manager.cc:1397] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.173s user 0.023s sys 0.030s
I20250112 20:51:48.898018 1164 raft_consensus.cc:357] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.898280 1164 raft_consensus.cc:383] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.900192 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 532ccaeaf38c41a6ab26f65411d31418
W20250112 20:51:48.904127 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c6bdfecd5d9142d5b008c613864c8217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bdfecd5d9142d5b008c613864c8217 already in progress: creating tablet
W20250112 20:51:48.904430 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f0fb6a5146c44edea076a10da58ea1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fb6a5146c44edea076a10da58ea1a7 already in progress: creating tablet
I20250112 20:51:48.905738 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cb6c84ab3dd344d28800d498d1816591" } new_table_name: "default.Table_394111b499bd4b8ca206c37121264bb4" modify_external_catalogs: true
W20250112 20:51:48.906370 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 53338.7us
I20250112 20:51:48.906723 1164 raft_consensus.cc:738] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.906940 1164 consensus_queue.cc:260] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.907447 1164 ts_tablet_manager.cc:1428] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.010s user 0.000s sys 0.000s
W20250112 20:51:48.903636 710 leader_election.cc:343] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.907397 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f568569d7234222a4ea47ebca7c2170"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.907622 710 leader_election.cc:304] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:48.908018 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f568569d7234222a4ea47ebca7c2170 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.908206 2923 raft_consensus.cc:2798] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.908277 2923 raft_consensus.cc:491] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.908356 2923 raft_consensus.cc:3054] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.909551 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 45484f8e48524b86a6f418b88a71ac18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.909986 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1e9b4851670045b1975a05cdd3f5c2f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.910436 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e9b4851670045b1975a05cdd3f5c2f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.910847 2923 raft_consensus.cc:513] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.911372 2923 leader_election.cc:290] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.911598 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_137d3f9986ea49519c9585b3761b586c" } modify_external_catalogs: true
I20250112 20:51:48.911769 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_137d3f9986ea49519c9585b3761b586c" } modify_external_catalogs: true
I20250112 20:51:48.911971 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.912012 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.912117 764 raft_consensus.cc:2463] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.912191 771 raft_consensus.cc:3054] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.912288 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.912400 760 raft_consensus.cc:3054] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.912909 774 raft_consensus.cc:3054] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.913100 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3fae1e5ce26f4ad593ad310c09385e88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:48.913578 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fae1e5ce26f4ad593ad310c09385e88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.914978 760 raft_consensus.cc:2463] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.915746 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_162669e16db04dd4a6ea5aa5fd553e4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:48.915937 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.916077 767 raft_consensus.cc:2463] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
W20250112 20:51:48.916190 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_162669e16db04dd4a6ea5aa5fd553e4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:48.916276 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4a237cfeeb4d0fb7ad1abb4ff97845" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.916437 768 raft_consensus.cc:2463] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:48.917508 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.917670 766 raft_consensus.cc:3054] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.920595 766 raft_consensus.cc:2463] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.924091 771 raft_consensus.cc:2463] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.924741 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4a237cfeeb4d0fb7ad1abb4ff97845" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.924909 772 raft_consensus.cc:3054] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:48.925051 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f424288fe62849edbbd2d38e89331fb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424288fe62849edbbd2d38e89331fb7 already in progress: creating tablet
W20250112 20:51:48.925351 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6158b97ccf0a42fdb1a510c5d1e80213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6158b97ccf0a42fdb1a510c5d1e80213 already in progress: creating tablet
I20250112 20:51:48.925805 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (table default.Table_0787fb4223774e1889d1d09fb80f87f5 [id=fb7c117897c24636b7e2dd8109e782d2]) successfully deleted
I20250112 20:51:48.926045 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1fd34e829d0645ea80daaff5d27cb266 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
I20250112 20:51:48.926221 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1fd47d1c1a4b42bdbff97f619dc95134 (table default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]) successfully deleted
W20250112 20:51:48.926445 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e8bea4d83ea644a0930a29f61e7a5fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8bea4d83ea644a0930a29f61e7a5fc8 already in progress: creating tablet
I20250112 20:51:48.927127 772 raft_consensus.cc:2463] T be4a237cfeeb4d0fb7ad1abb4ff97845 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:48.927291 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 (DEFAULT_TABLE table=default.Table_4a7a33e2dd7b42ada35718df1413bdd4 [id=70aa77642969417ca31d9b112034370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.927772 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096ee7d3a66846f9b619d8f2fc046616. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.927906 1164 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:48.929563 1164 tablet_bootstrap.cc:654] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.931610 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f875f075e98e45499291b519caf03184" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.931834 765 raft_consensus.cc:2388] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.933113 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.933343 765 raft_consensus.cc:2388] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87ae45957c64469d96dc92d606b5b4b3 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.933701 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:48.935442 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:48.935601 765 raft_consensus.cc:2463] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:48.936024 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9989f3dc61df4f75b9bdb85ba3ca3f37" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:48.936170 765 raft_consensus.cc:3054] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.938779 765 raft_consensus.cc:2463] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:48.938995 878 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc63eff44414906a027a8208b5cc5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.939270 888 tablet_service.cc:1514] Processing DeleteTablet for tablet ed925a5ef3a54f819ca292a4e7967b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.939440 889 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.939826 888 tablet_service.cc:1467] Processing CreateTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 (DEFAULT_TABLE table=default.Table_137d3f9986ea49519c9585b3761b586c [id=e697ab0e40864bd49fbc206de2bbf227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.940193 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7959f3ed319f4078ab747d490dce7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.940266 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3846863c74f64923bda6e5a19e7f2770 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:48.942461 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.943470 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.944650 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7972856872e7411e9a3651b4413ac951" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.944903 909 raft_consensus.cc:2388] T 7972856872e7411e9a3651b4413ac951 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.946004 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e150c031e4f4e99966c14e9005d00" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.946231 907 raft_consensus.cc:2388] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
I20250112 20:51:48.947120 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.948977 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea98c29bc744009ab9b3eeb53684f29" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.949241 907 raft_consensus.cc:2388] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aefe1db8b19244d891b6bcd8971e9b17 in current term 1: Already voted for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in this term.
W20250112 20:51:48.958166 847 leader_election.cc:343] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.958385 847 leader_election.cc:304] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:48.958464 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d94c67881e954c0ebecf5c31de665ac0 consensus state has no leader
I20250112 20:51:48.958710 2975 raft_consensus.cc:2743] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:48.960815 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:48.960985 907 raft_consensus.cc:2463] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:48.964306 753 tablet_service.cc:1467] Processing CreateTablet for tablet ead7549130fd431daa5ef43553f31792 (DEFAULT_TABLE table=default.Table_8bea75e61b954ec8b0838e2d6fa87628 [id=3b7b2127e4084624bed7b916b352bdfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.964569 3077 raft_consensus.cc:491] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.964514 1012 tablet_service.cc:1467] Processing CreateTablet for tablet ead7549130fd431daa5ef43553f31792 (DEFAULT_TABLE table=default.Table_8bea75e61b954ec8b0838e2d6fa87628 [id=3b7b2127e4084624bed7b916b352bdfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.964689 3077 raft_consensus.cc:513] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.964888 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead7549130fd431daa5ef43553f31792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.964934 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead7549130fd431daa5ef43553f31792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.965296 3077 leader_election.cc:290] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.966063 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 943) took 2449 ms. Trace:
I20250112 20:51:48.966137 1186 rpcz_store.cc:276] 0112 20:51:46.516712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.516844 (+ 132us) service_pool.cc:225] Handling call
0112 20:51:48.839153 (+2322309us) ts_tablet_manager.cc:1145] Deleting tablet 7f807071e53a4f0e8629b1dfa6d26b9d
0112 20:51:48.839164 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.874516 (+ 35352us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.877353 (+ 2837us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:48.966049 (+ 88696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2321154}
I20250112 20:51:48.966399 1186 tablet_replica.cc:331] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:48.964516 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:48.969939 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:48.970098 907 raft_consensus.cc:3054] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.970561 1186 raft_consensus.cc:2238] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:48.970686 1186 raft_consensus.cc:2267] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:48.971140 1186 ts_tablet_manager.cc:1905] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:48.971931 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e584b761d234c5aa0ab875acd8e1e05" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:48.972121 1039 raft_consensus.cc:3054] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.972559 907 raft_consensus.cc:2463] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.975227 3077 raft_consensus.cc:491] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.975430 1039 raft_consensus.cc:2463] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:48.975373 3077 raft_consensus.cc:513] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:48.975924 3077 leader_election.cc:290] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:48.978116 2787 consensus_queue.cc:1035] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.980010 2787 consensus_queue.cc:1035] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.981362 2964 consensus_queue.cc:1035] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.982126 2964 consensus_queue.cc:1035] T 0093059b34fb40609289f578978c8750 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.983191 2968 consensus_queue.cc:1035] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.986848 447 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
W20250112 20:51:48.987139 983 consensus_peers.cc:487] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fd34e829d0645ea80daaff5d27cb266. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:48.991892 2964 consensus_queue.cc:1035] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.994459 752 tablet_service.cc:1467] Processing CreateTablet for tablet c19c8965b19241aeadc4accc8205b9c5 (DEFAULT_TABLE table=default.Table_d62facea3c274e3ca3bc123e1402be3a [id=df5bb5f118a14f13a3a7a49b795c85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.994777 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d8bf77c52fe545e08835a1db891d15a1 (DEFAULT_TABLE table=default.Table_1bc274860e8247998efa842371c1f86a [id=cd69da92b24f4a21914bcbfc50783b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:48.994971 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c8965b19241aeadc4accc8205b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.995364 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf77c52fe545e08835a1db891d15a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:48.996301 2975 raft_consensus.cc:491] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.996419 2975 raft_consensus.cc:513] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:48.996971 2975 leader_election.cc:290] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:48.997629 847 leader_election.cc:343] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:48.997776 847 leader_election.cc:304] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:48.998188 2975 raft_consensus.cc:2743] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.001744 744 tablet_service.cc:1467] Processing CreateTablet for tablet f553239e6a9240eabaaaf3431b4a65b6 (DEFAULT_TABLE table=default.Table_dd1d3864738a483d917f02e276a04376 [id=ac4af431ce6d496fbbcd732b2163588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.002043 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c19c8965b19241aeadc4accc8205b9c5 (DEFAULT_TABLE table=default.Table_d62facea3c274e3ca3bc123e1402be3a [id=df5bb5f118a14f13a3a7a49b795c85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.002179 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f553239e6a9240eabaaaf3431b4a65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.002452 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c8965b19241aeadc4accc8205b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.017586 3085 tablet.cc:1721] T 5ccb593413124921af94d0b855789b52 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:49.020620 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f553239e6a9240eabaaaf3431b4a65b6 (DEFAULT_TABLE table=default.Table_dd1d3864738a483d917f02e276a04376 [id=ac4af431ce6d496fbbcd732b2163588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.021118 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f553239e6a9240eabaaaf3431b4a65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.021189 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7959f3ed319f4078ab747d490dce7fe1
I20250112 20:51:49.025223 3078 tablet.cc:1721] T d477fb5e5ad5447bb182bc9998561a43 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.026113 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1596ecc13e4a4de58315a28d04697a1b (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.026348 1022 tablet_service.cc:1467] Processing CreateTablet for tablet de6ae0d6a6fc4202aea20699d2548362 (DEFAULT_TABLE table=default.Table_ee6cb2e8069b41899759a8b3c283152f [id=1679c8d02902418cb0e2b97442da4e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.026492 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1596ecc13e4a4de58315a28d04697a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.026754 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6ae0d6a6fc4202aea20699d2548362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.036362 847 leader_election.cc:304] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:49.036648 847 raft_consensus.cc:2681] T 7f807071e53a4f0e8629b1dfa6d26b9d P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:49.036850 710 leader_election.cc:304] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:49.037390 3077 raft_consensus.cc:2743] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.037542 2447 ts_tablet_manager.cc:1918] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:49.037647 2447 log.cc:1198] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3343e1d7f1d04a498bd0ee00311b6a61
I20250112 20:51:49.037595 710 leader_election.cc:304] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:49.037995 2447 ts_tablet_manager.cc:1939] T 3343e1d7f1d04a498bd0ee00311b6a61 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.037993 3077 raft_consensus.cc:2798] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.038085 3077 raft_consensus.cc:491] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.038195 3077 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.038444 2954 consensus_queue.cc:1035] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.039656 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1596ecc13e4a4de58315a28d04697a1b (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.040141 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1596ecc13e4a4de58315a28d04697a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.040613 3077 raft_consensus.cc:513] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.041239 3077 leader_election.cc:290] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.049216 1164 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.049389 1164 ts_tablet_manager.cc:1397] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.005s
I20250112 20:51:49.050338 752 tablet_service.cc:1467] Processing CreateTablet for tablet de6ae0d6a6fc4202aea20699d2548362 (DEFAULT_TABLE table=default.Table_ee6cb2e8069b41899759a8b3c283152f [id=1679c8d02902418cb0e2b97442da4e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.050400 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 2e163a5b99a44d34bf59ada02c2bba99 (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.050820 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6ae0d6a6fc4202aea20699d2548362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.050846 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e163a5b99a44d34bf59ada02c2bba99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.056558 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2e163a5b99a44d34bf59ada02c2bba99 (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.057209 753 tablet_service.cc:1467] Processing CreateTablet for tablet d8bf77c52fe545e08835a1db891d15a1 (DEFAULT_TABLE table=default.Table_1bc274860e8247998efa842371c1f86a [id=cd69da92b24f4a21914bcbfc50783b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.057425 2968 consensus_queue.cc:1035] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.057806 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e163a5b99a44d34bf59ada02c2bba99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.058840 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 93c40c91021f4e5998c9acca3a8231e8 (DEFAULT_TABLE table=default.Table_0d68049237d04746aed93d5313f67bc0 [id=3a23c58c86f34a07856f8fef7cd4011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.059262 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf77c52fe545e08835a1db891d15a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.059340 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c40c91021f4e5998c9acca3a8231e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.063256 1164 raft_consensus.cc:357] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.063458 1164 raft_consensus.cc:383] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.063544 1164 raft_consensus.cc:738] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.063791 1164 consensus_queue.cc:260] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.063894 3086 tablet.cc:1721] T d477fb5e5ad5447bb182bc9998561a43 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.065117 1186 ts_tablet_manager.cc:1918] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:49.065310 1186 log.cc:1198] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/924082716c1e4d9ea6fbe97ae93232bd
I20250112 20:51:49.065680 1186 ts_tablet_manager.cc:1939] T 924082716c1e4d9ea6fbe97ae93232bd P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:49.066444 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 945) took 2549 ms. Trace:
I20250112 20:51:49.067193 1164 ts_tablet_manager.cc:1428] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20250112 20:51:49.067519 3077 consensus_queue.cc:1035] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.067674 1164 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.068861 1186 rpcz_store.cc:276] 0112 20:51:46.517209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.517253 (+ 44us) service_pool.cc:225] Handling call
0112 20:51:48.966363 (+2449110us) ts_tablet_manager.cc:1145] Deleting tablet 924082716c1e4d9ea6fbe97ae93232bd
0112 20:51:48.966371 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.030842 (+ 64471us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.065019 (+ 34177us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.066426 (+ 1407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2448309}
I20250112 20:51:49.069552 1164 tablet_bootstrap.cc:654] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.069800 1186 tablet_replica.cc:331] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:49.072742 3078 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47160 (request call id 998) took 1074 ms. Trace:
I20250112 20:51:49.077327 745 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 (DEFAULT_TABLE table=default.Table_16215b3f2b9d4bba913676a2f2590409 [id=d5c16096a83649aa9cc0794b5055c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.078217 742 tablet_service.cc:1467] Processing CreateTablet for tablet 08b293868ae546c88f393b17e51147d0 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.078776 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2160013496154cbb956102ec8d68e3bb (DEFAULT_TABLE table=default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391 [id=fefa5912a94b4c0ca7a905c73556599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.079433 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.078209 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a4ec67af024fe0be2c06e35117acc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.084406 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2160013496154cbb956102ec8d68e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.084559 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 (DEFAULT_TABLE table=default.Table_16215b3f2b9d4bba913676a2f2590409 [id=d5c16096a83649aa9cc0794b5055c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.085369 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 08b293868ae546c88f393b17e51147d0 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.087432 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 2160013496154cbb956102ec8d68e3bb (DEFAULT_TABLE table=default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391 [id=fefa5912a94b4c0ca7a905c73556599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.087713 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.087853 1164 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.087983 1164 ts_tablet_manager.cc:1397] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20250112 20:51:49.088400 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b293868ae546c88f393b17e51147d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.088670 847 leader_election.cc:304] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:49.088667 1164 raft_consensus.cc:357] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.088881 1164 raft_consensus.cc:383] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.088953 1164 raft_consensus.cc:738] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.090283 1164 consensus_queue.cc:260] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.090718 1185 ts_tablet_manager.cc:1918] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.090847 1185 log.cc:1198] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8229e4ef771b4032a4bfae3190a851e3
I20250112 20:51:49.091236 1185 ts_tablet_manager.cc:1939] T 8229e4ef771b4032a4bfae3190a851e3 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:49.092555 1164 ts_tablet_manager.cc:1428] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:49.095475 2983 raft_consensus.cc:2743] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:49.096237 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 61964ec077094c018c5a03ad7699c9c6 (table default.Table_a1de4aac1b4f453dbbd50718c2d98dcf [id=e8f7a569b2914e54b0819c8387667edf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 61964ec077094c018c5a03ad7699c9c6 consensus state has no leader
I20250112 20:51:49.097201 847 leader_election.cc:304] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:49.097498 2983 raft_consensus.cc:2743] T fe7e150c031e4f4e99966c14e9005d00 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.099356 2983 consensus_queue.cc:1035] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.100239 847 leader_election.cc:304] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.100582 2983 raft_consensus.cc:2798] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.100664 2983 raft_consensus.cc:491] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.100740 2983 raft_consensus.cc:3054] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.100751 710 leader_election.cc:304] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.101212 3077 raft_consensus.cc:2798] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.101343 3077 raft_consensus.cc:491] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.101431 3077 raft_consensus.cc:3054] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.103521 2983 raft_consensus.cc:513] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.103701 3077 raft_consensus.cc:513] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.101253 710 leader_election.cc:304] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.104200 2983 leader_election.cc:290] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.104239 3077 leader_election.cc:290] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.104450 3077 raft_consensus.cc:2798] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.104748 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a4ec67af024fe0be2c06e35117acc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.105227 3077 raft_consensus.cc:695] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.105837 2923 raft_consensus.cc:491] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.105978 2923 raft_consensus.cc:513] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.106185 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f264d84ae4846d1afaaabc41d5789f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.106353 907 raft_consensus.cc:2463] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:49.106467 2923 leader_election.cc:290] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.105509 3077 consensus_queue.cc:237] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.107755 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.107810 710 leader_election.cc:304] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:49.107929 909 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.108064 2954 raft_consensus.cc:3054] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.109004 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.110471 909 raft_consensus.cc:2463] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:49.110499 2954 raft_consensus.cc:2743] T 7972856872e7411e9a3651b4413ac951 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.110721 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.110862 905 raft_consensus.cc:3054] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.110998 710 leader_election.cc:304] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:49.111487 2923 raft_consensus.cc:2743] T fe7e150c031e4f4e99966c14e9005d00 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.111703 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2160013496154cbb956102ec8d68e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.111629 2632 consensus_queue.cc:1035] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.111699 2923 consensus_queue.cc:1035] T fa2f3389061143509354f37ae93ef0c7 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.112326 710 leader_election.cc:304] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:49.112532 2632 raft_consensus.cc:2743] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.113018 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b293868ae546c88f393b17e51147d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.113221 905 raft_consensus.cc:2463] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:49.114042 710 leader_election.cc:343] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.116990 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d6b4effb84314a6b2b9db883b4366" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.117179 905 raft_consensus.cc:2463] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:49.118228 710 leader_election.cc:304] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.118494 2632 raft_consensus.cc:2798] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.118623 2632 raft_consensus.cc:491] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.118724 2632 raft_consensus.cc:3054] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.121905 2632 raft_consensus.cc:513] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.122709 2632 leader_election.cc:290] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.123046 710 leader_election.cc:304] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.123446 710 leader_election.cc:304] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.123842 2886 raft_consensus.cc:2798] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.123642 2632 raft_consensus.cc:2798] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.123983 2886 raft_consensus.cc:695] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.124307 2886 consensus_queue.cc:237] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.129021 2632 raft_consensus.cc:695] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.129594 2632 consensus_queue.cc:237] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.129982 710 leader_election.cc:304] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.131058 2954 raft_consensus.cc:2798] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.131168 2954 raft_consensus.cc:491] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.131263 2954 raft_consensus.cc:3054] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.134218 2954 raft_consensus.cc:513] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.134867 2954 leader_election.cc:290] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.076465 3078 rpcz_store.cc:276] 0112 20:51:47.998573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.998612 (+ 39us) service_pool.cc:225] Handling call
0112 20:51:49.072724 (+1074112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:48.006125 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 20:51:48.006220 (+ 95us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 20:51:48.006231 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0112 20:51:48.006239 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 20:51:48.006279 (+ 40us) alter_schema_op.cc:128] START. Timestamp: 1736715108006227
0112 20:51:48.006284 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 20:51:48.006463 (+ 179us) log.cc:844] Serialized 173 byte log entry
0112 20:51:48.888606 (+882143us) op_driver.cc:464] REPLICATION: finished
0112 20:51:49.025197 (+136591us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 20:51:49.025617 (+ 420us) log.cc:844] Serialized 5 byte log entry
0112 20:51:49.071501 (+ 45884us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.072178 (+ 677us) log.cc:844] Serialized 14 byte log entry
0112 20:51:49.072686 (+ 508us) alter_schema_op.cc:73] Released schema lock
0112 20:51:49.072691 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":136531,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":459,"prepare.run_wall_time_us":458,"raft.queue_time_us":427,"raft.run_cpu_time_us":474,"raft.run_wall_time_us":475,"replication_time_us":882289,"thread_start_us":179297,"threads_started":2,"wal-append.queue_time_us":34153}]]}
I20250112 20:51:49.135658 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f264d84ae4846d1afaaabc41d5789f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.135859 905 raft_consensus.cc:3054] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.139106 905 raft_consensus.cc:2463] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:49.140229 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d6b4effb84314a6b2b9db883b4366" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.140473 905 raft_consensus.cc:3054] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.143761 710 leader_election.cc:304] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.146202 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd4177f9394a5a82d4f5e02369b3b6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.146288 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f264d84ae4846d1afaaabc41d5789f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.144371 752 tablet_service.cc:1467] Processing CreateTablet for tablet 93c40c91021f4e5998c9acca3a8231e8 (DEFAULT_TABLE table=default.Table_0d68049237d04746aed93d5313f67bc0 [id=3a23c58c86f34a07856f8fef7cd4011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.146960 2954 raft_consensus.cc:2798] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.147251 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.147397 2954 raft_consensus.cc:695] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.147714 2954 consensus_queue.cc:237] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.148717 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.149696 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d6b4effb84314a6b2b9db883b4366" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.151727 905 raft_consensus.cc:2463] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:49.152331 710 leader_election.cc:304] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.152601 2923 raft_consensus.cc:2798] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:49.157433 885 consensus_meta.cc:287] T 78b166a5954945d0a6bb167ba68cc19a P 87ae45957c64469d96dc92d606b5b4b3: Time spent flushing consensus metadata: real 0.874s user 0.025s sys 0.000s
I20250112 20:51:49.162556 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 932) took 1615 ms. Trace:
I20250112 20:51:49.162766 2447 rpcz_store.cc:276] 0112 20:51:47.546813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.546861 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:48.745611 (+1198750us) ts_tablet_manager.cc:1145] Deleting tablet 3343e1d7f1d04a498bd0ee00311b6a61
0112 20:51:48.745621 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.869271 (+123650us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.037452 (+168181us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.162531 (+125079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198574}
I20250112 20:51:49.163298 2447 tablet_replica.cc:331] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:49.163697 2447 raft_consensus.cc:2238] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.163951 2447 raft_consensus.cc:2267] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.164654 2447 ts_tablet_manager.cc:1905] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.165371 2983 raft_consensus.cc:491] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.165522 2983 raft_consensus.cc:513] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.166249 2983 leader_election.cc:290] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.176553 2923 raft_consensus.cc:695] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.177043 2923 consensus_queue.cc:237] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.185003 2968 consensus_queue.cc:1035] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.186358 2954 consensus_queue.cc:1035] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.198165 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.216487 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f264d84ae4846d1afaaabc41d5789f4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.216902 710 leader_election.cc:304] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:49.216904 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104d6b4effb84314a6b2b9db883b4366" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:49.217480 710 leader_election.cc:343] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.220180 2923 raft_consensus.cc:2743] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:49.224324 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c40c91021f4e5998c9acca3a8231e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.228052 2964 consensus_queue.cc:1035] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.228890 2964 consensus_queue.cc:1035] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.229665 2964 consensus_queue.cc:1035] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.230464 2964 consensus_queue.cc:1035] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.232409 2964 consensus_queue.cc:1035] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.253021 1186 raft_consensus.cc:2238] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.253261 1186 raft_consensus.cc:2267] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.261212 1186 ts_tablet_manager.cc:1905] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.268103 1164 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.270087 1164 tablet_bootstrap.cc:654] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.276079 1163 tablet_bootstrap.cc:654] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.277778 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 3343e1d7f1d04a498bd0ee00311b6a61 (table default.Table_22041bd544d842dd8bba1f206c270a4e [id=1e2fe154afd64f478a3d61795fc95047]) successfully deleted
I20250112 20:51:49.279772 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 01367a11dea04b33a9255508501b9060 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:45 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.280196 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:49.280911 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 180677us
I20250112 20:51:49.281096 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.281215 847 leader_election.cc:304] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.281553 2983 raft_consensus.cc:2798] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.281685 2983 raft_consensus.cc:695] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:49.281988 2983 consensus_queue.cc:237] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.283056 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d97ac93467a4a7a8ae44d52719f191c with delete_type TABLET_DATA_DELETED (replaced by tablet f52a9216c5044f88a0060e3e8e95eddd) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.290076 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 953) took 2580 ms. Trace:
I20250112 20:51:49.290196 1185 rpcz_store.cc:276] 0112 20:51:46.709473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.710434 (+ 961us) service_pool.cc:225] Handling call
0112 20:51:48.765654 (+2055220us) ts_tablet_manager.cc:1145] Deleting tablet 8229e4ef771b4032a4bfae3190a851e3
0112 20:51:48.765666 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:48.891056 (+125390us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.090586 (+199530us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.290027 (+199441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055039}
I20250112 20:51:49.290810 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 954) took 2580 ms. Trace:
I20250112 20:51:49.290951 1185 rpcz_store.cc:276] 0112 20:51:46.710477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.710646 (+ 169us) service_pool.cc:225] Handling call
0112 20:51:49.290667 (+2580021us) ts_tablet_manager.cc:1145] Deleting tablet 2de03e3114d145aeb81590dbae2c548e
0112 20:51:49.290677 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.290777 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2579902}
I20250112 20:51:49.292006 3097 tablet.cc:1721] T d477fb5e5ad5447bb182bc9998561a43 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.307464 1186 ts_tablet_manager.cc:1918] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:49.307602 1186 log.cc:1198] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/dc3a9740415140fe8864d54f477af954
I20250112 20:51:49.307953 1186 ts_tablet_manager.cc:1939] T dc3a9740415140fe8864d54f477af954 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:49.311640 1185 tablet_replica.cc:331] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:49.311888 1185 raft_consensus.cc:2238] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:49.312158 1185 pending_rounds.cc:62] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:49.311801 451 catalog_manager.cc:5526] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:49.312245 1185 pending_rounds.cc:69] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585084338020352 op_type: NO_OP noop_request { }
I20250112 20:51:49.312361 1185 raft_consensus.cc:2883] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:49.312489 1185 raft_consensus.cc:2267] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.312501 451 catalog_manager.cc:5526] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:49.313199 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:49.313593 451 catalog_manager.cc:5526] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:49.314369 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 1e584b761d234c5aa0ab875acd8e1e05 (table default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a]). Expected version 4 got 0
I20250112 20:51:49.314618 1164 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.314739 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:49.314777 1164 ts_tablet_manager.cc:1397] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.006s
I20250112 20:51:49.315382 1164 raft_consensus.cc:357] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.315562 1164 raft_consensus.cc:383] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.315634 1164 raft_consensus.cc:738] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.315871 1164 consensus_queue.cc:260] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.316359 1164 ts_tablet_manager.cc:1428] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:49.316617 1164 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.318361 1164 tablet_bootstrap.cc:654] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:49.321992 847 leader_election.cc:343] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.324209 885 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57104 (request call id 1006) took 1047 ms. Trace:
I20250112 20:51:49.324345 885 rpcz_store.cc:276] 0112 20:51:48.277012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.278606 (+ 1594us) service_pool.cc:225] Handling call
0112 20:51:48.279248 (+ 642us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 20:51:48.279283 (+ 35us) ts_tablet_manager.cc:676] Creating new metadata...
0112 20:51:48.283546 (+ 4263us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.157695 (+874149us) tablet_replica.cc:206] Creating consensus instance
0112 20:51:49.324192 (+166497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 20:51:49.328184 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:49.328868 2983 consensus_queue.cc:1035] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.330106 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bea4d83ea644a0930a29f61e7a5fc8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:49.330312 763 raft_consensus.cc:2463] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:49.333468 888 tablet_service.cc:1467] Processing CreateTablet for tablet ead7549130fd431daa5ef43553f31792 (DEFAULT_TABLE table=default.Table_8bea75e61b954ec8b0838e2d6fa87628 [id=3b7b2127e4084624bed7b916b352bdfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.335618 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead7549130fd431daa5ef43553f31792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.338364 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.339445 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:49.339731 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:49.340126 450 catalog_manager.cc:5526] T 557fa6445eee481dbc4076f67fe63466 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } }
I20250112 20:51:49.333629 877 tablet_service.cc:1467] Processing CreateTablet for tablet d8bf77c52fe545e08835a1db891d15a1 (DEFAULT_TABLE table=default.Table_1bc274860e8247998efa842371c1f86a [id=cd69da92b24f4a21914bcbfc50783b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.341228 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf77c52fe545e08835a1db891d15a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.342223 1185 ts_tablet_manager.cc:1905] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.333779 889 tablet_service.cc:1467] Processing CreateTablet for tablet f553239e6a9240eabaaaf3431b4a65b6 (DEFAULT_TABLE table=default.Table_dd1d3864738a483d917f02e276a04376 [id=ac4af431ce6d496fbbcd732b2163588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.345180 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f553239e6a9240eabaaaf3431b4a65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.345584 1178 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:49.345731 1178 ts_tablet_manager.cc:1397] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.491s user 0.096s sys 0.130s
I20250112 20:51:49.346946 1178 raft_consensus.cc:357] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.347121 1178 raft_consensus.cc:383] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.347183 1178 raft_consensus.cc:738] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.347388 1178 consensus_queue.cc:260] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.347527 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:49.347690 772 raft_consensus.cc:3054] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.333779 878 tablet_service.cc:1467] Processing CreateTablet for tablet c19c8965b19241aeadc4accc8205b9c5 (DEFAULT_TABLE table=default.Table_d62facea3c274e3ca3bc123e1402be3a [id=df5bb5f118a14f13a3a7a49b795c85c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.349824 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c8965b19241aeadc4accc8205b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.350759 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:49.352089 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 98547168378947df885c601ccdc85e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.352766 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dba3d9315db4b9292288ee62c2244bc with delete_type TABLET_DATA_DELETED (replaced by tablet 3846863c74f64923bda6e5a19e7f2770) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.352984 742 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc22207e9204b73877c74f61ebba8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.333956 884 tablet_service.cc:1467] Processing CreateTablet for tablet de6ae0d6a6fc4202aea20699d2548362 (DEFAULT_TABLE table=default.Table_ee6cb2e8069b41899759a8b3c283152f [id=1679c8d02902418cb0e2b97442da4e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.353440 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6ae0d6a6fc4202aea20699d2548362. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.354712 710 leader_election.cc:343] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.354758 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40e45cdb124965b70e054ec37ddee0 (DEFAULT_TABLE table=default.Table_7561a010074e4f19ba6a480d869bb939 [id=530ccbbd1b9c4f09bbee4ea8ecb20d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.355135 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c40e45cdb124965b70e054ec37ddee0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.355172 710 leader_election.cc:343] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.355556 710 leader_election.cc:343] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.355836 710 leader_election.cc:343] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.356137 710 leader_election.cc:343] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.356276 887 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa2cdfd1fa460eb0a997d6750c036d (DEFAULT_TABLE table=default.Table_688c94cdb32f468587b57b9c7e9a895c [id=dab4e3908e224559a4ff801e128520a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.356587 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0aa2cdfd1fa460eb0a997d6750c036d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.357079 747 tablet_service.cc:1467] Processing CreateTablet for tablet aeb83d80e48b4f4081920ffb0f121f2e (DEFAULT_TABLE table=default.Table_78ebfe3d8c41485999992166d9bf2149 [id=f954d6b123834e8ab7627c3318e7570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.357460 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb83d80e48b4f4081920ffb0f121f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.334122 875 tablet_service.cc:1467] Processing CreateTablet for tablet 1596ecc13e4a4de58315a28d04697a1b (DEFAULT_TABLE table=default.Table_63b6aefc10aa40f7b6b171b2687a1bc2 [id=c862be87d9484763aaa3142a357ef099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.362872 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1596ecc13e4a4de58315a28d04697a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.363466 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.334300 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2e163a5b99a44d34bf59ada02c2bba99 (DEFAULT_TABLE table=default.Table_f7662b40ae1447808abeaa1501705e2b [id=534a928db1f544818959f01ad583f7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.365113 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e163a5b99a44d34bf59ada02c2bba99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.328063 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 96663.6us
I20250112 20:51:49.336690 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.346217 1040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bea4d83ea644a0930a29f61e7a5fc8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.365983 1040 raft_consensus.cc:2463] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:49.350474 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.352171 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.364123 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40e45cdb124965b70e054ec37ddee0 (DEFAULT_TABLE table=default.Table_7561a010074e4f19ba6a480d869bb939 [id=530ccbbd1b9c4f09bbee4ea8ecb20d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.366664 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c40e45cdb124965b70e054ec37ddee0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.367551 983 consensus_peers.cc:487] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:49.367699 983 consensus_peers.cc:487] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f807071e53a4f0e8629b1dfa6d26b9d. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:49.334468 892 tablet_service.cc:1467] Processing CreateTablet for tablet 93c40c91021f4e5998c9acca3a8231e8 (DEFAULT_TABLE table=default.Table_0d68049237d04746aed93d5313f67bc0 [id=3a23c58c86f34a07856f8fef7cd4011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.368604 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c40c91021f4e5998c9acca3a8231e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.364672 1010 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa2cdfd1fa460eb0a997d6750c036d (DEFAULT_TABLE table=default.Table_688c94cdb32f468587b57b9c7e9a895c [id=dab4e3908e224559a4ff801e128520a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.369102 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0aa2cdfd1fa460eb0a997d6750c036d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.371536 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_0a07af8986ee4235a0240a69d577e2c4 [id=7ac468f9c41243d091dee834d9f2be49] alter complete (version 2)
I20250112 20:51:49.334631 891 tablet_service.cc:1467] Processing CreateTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 (DEFAULT_TABLE table=default.Table_16215b3f2b9d4bba913676a2f2590409 [id=d5c16096a83649aa9cc0794b5055c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.373131 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a4ec67af024fe0be2c06e35117acc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.374977 2787 consensus_queue.cc:1035] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.375543 2787 consensus_queue.cc:1035] T 7972856872e7411e9a3651b4413ac951 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.334743 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.334800 876 tablet_service.cc:1467] Processing CreateTablet for tablet 08b293868ae546c88f393b17e51147d0 (DEFAULT_TABLE table=default.Table_2783ab15d9c84d9a9b09c3688bce6109 [id=c39eb08b91644c65b6ac915a1ba356eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.377293 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b293868ae546c88f393b17e51147d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.378340 1164 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.378459 1164 ts_tablet_manager.cc:1397] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.002s
I20250112 20:51:49.379108 1164 raft_consensus.cc:357] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.379274 1164 raft_consensus.cc:383] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.379328 1164 raft_consensus.cc:738] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.379504 1164 consensus_queue.cc:260] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.380402 745 tablet_service.cc:1467] Processing CreateTablet for tablet c3812f8aa9ba422f85f331648a1ec02e (DEFAULT_TABLE table=default.Table_bdcb34e5580647dba61719814e3ce733 [id=3c0f9fd2807646e09df8875e8433f90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.381098 738 tablet_service.cc:1467] Processing CreateTablet for tablet c17e8808d62d4b7da88bffe7d7569568 (DEFAULT_TABLE table=default.Table_eba10319e22144a4bb6eee085519a0fb [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.381319 880 tablet_service.cc:1467] Processing CreateTablet for tablet ced81345e31a4634bb21e3702ad69c93 (DEFAULT_TABLE table=default.Table_ae66395240e9403a94131921dfe99d18 [id=a7f0bc0c9fce44638273cfdcf3c68d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.381599 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced81345e31a4634bb21e3702ad69c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.334959 874 tablet_service.cc:1467] Processing CreateTablet for tablet 2160013496154cbb956102ec8d68e3bb (DEFAULT_TABLE table=default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391 [id=fefa5912a94b4c0ca7a905c73556599a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.382020 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2160013496154cbb956102ec8d68e3bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.383235 1028 tablet_service.cc:1467] Processing CreateTablet for tablet aeb83d80e48b4f4081920ffb0f121f2e (DEFAULT_TABLE table=default.Table_78ebfe3d8c41485999992166d9bf2149 [id=f954d6b123834e8ab7627c3318e7570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.383550 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb83d80e48b4f4081920ffb0f121f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.335091 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 532ccaeaf38c41a6ab26f65411d31418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.335137 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.335206 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.335017 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0866871b9e1542e8ab30505c68f7dea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.387481 887 tablet_service.cc:1467] Processing CreateTablet for tablet c3812f8aa9ba422f85f331648a1ec02e (DEFAULT_TABLE table=default.Table_bdcb34e5580647dba61719814e3ce733 [id=3c0f9fd2807646e09df8875e8433f90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.387800 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3812f8aa9ba422f85f331648a1ec02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.389386 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.389768 881 tablet_service.cc:1467] Processing CreateTablet for tablet aeb83d80e48b4f4081920ffb0f121f2e (DEFAULT_TABLE table=default.Table_78ebfe3d8c41485999992166d9bf2149 [id=f954d6b123834e8ab7627c3318e7570f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.390115 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb83d80e48b4f4081920ffb0f121f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.393107 879 tablet_service.cc:1467] Processing CreateTablet for tablet 7c40e45cdb124965b70e054ec37ddee0 (DEFAULT_TABLE table=default.Table_7561a010074e4f19ba6a480d869bb939 [id=530ccbbd1b9c4f09bbee4ea8ecb20d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.393468 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c40e45cdb124965b70e054ec37ddee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.395951 1163 tablet_bootstrap.cc:492] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.396101 1163 ts_tablet_manager.cc:1397] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.912s user 0.024s sys 0.000s
I20250112 20:51:49.396907 1163 raft_consensus.cc:357] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.397071 1163 raft_consensus.cc:383] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.397116 1163 raft_consensus.cc:738] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.357087 753 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa2cdfd1fa460eb0a997d6750c036d (DEFAULT_TABLE table=default.Table_688c94cdb32f468587b57b9c7e9a895c [id=dab4e3908e224559a4ff801e128520a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.397284 1163 consensus_queue.cc:260] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.397840 1163 ts_tablet_manager.cc:1428] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:49.398075 1163 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:49.398361 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3812f8aa9ba422f85f331648a1ec02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.399425 1163 tablet_bootstrap.cc:654] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.399737 1012 tablet_service.cc:1467] Processing CreateTablet for tablet ced81345e31a4634bb21e3702ad69c93 (DEFAULT_TABLE table=default.Table_ae66395240e9403a94131921dfe99d18 [id=a7f0bc0c9fce44638273cfdcf3c68d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.400063 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced81345e31a4634bb21e3702ad69c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.400980 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17e8808d62d4b7da88bffe7d7569568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.404160 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.404690 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaa7c75d0c548a987d5bf90f9f7000b (DEFAULT_TABLE table=default.Table_b7d3364377a4471b9d511e51e1467297 [id=0970bb7daa834c4aa04a460b7bc21835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.405025 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaa7c75d0c548a987d5bf90f9f7000b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.405552 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaa7c75d0c548a987d5bf90f9f7000b (DEFAULT_TABLE table=default.Table_b7d3364377a4471b9d511e51e1467297 [id=0970bb7daa834c4aa04a460b7bc21835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.406195 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaa7c75d0c548a987d5bf90f9f7000b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.408928 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.409315 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0aa2cdfd1fa460eb0a997d6750c036d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.410816 3128 consensus_queue.cc:1035] T 557fa6445eee481dbc4076f67fe63466 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.413762 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaa7c75d0c548a987d5bf90f9f7000b (DEFAULT_TABLE table=default.Table_b7d3364377a4471b9d511e51e1467297 [id=0970bb7daa834c4aa04a460b7bc21835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.414147 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaa7c75d0c548a987d5bf90f9f7000b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.417961 2983 raft_consensus.cc:491] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.418175 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7959f3ed319f4078ab747d490dce7fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.418114 2983 raft_consensus.cc:513] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.418707 2983 leader_election.cc:290] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.419829 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 944) took 2902 ms. Trace:
I20250112 20:51:49.419934 1186 rpcz_store.cc:276] 0112 20:51:46.516987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.517250 (+ 263us) service_pool.cc:225] Handling call
0112 20:51:49.069741 (+2552491us) ts_tablet_manager.cc:1145] Deleting tablet dc3a9740415140fe8864d54f477af954
0112 20:51:49.069764 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.267341 (+197577us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.307355 (+ 40014us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.419782 (+112427us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29893,"tablet-delete.queue_time_us":2551572}
I20250112 20:51:49.420308 1186 tablet_replica.cc:331] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:49.420524 1186 raft_consensus.cc:2238] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.420696 1186 raft_consensus.cc:2267] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.421599 1186 ts_tablet_manager.cc:1905] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.424638 3078 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.427909 1186 ts_tablet_manager.cc:1918] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.428850 1178 ts_tablet_manager.cc:1428] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20250112 20:51:49.429284 1178 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:49.430833 1164 ts_tablet_manager.cc:1428] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.052s user 0.005s sys 0.005s
I20250112 20:51:49.431102 1178 tablet_bootstrap.cc:654] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.431668 772 raft_consensus.cc:2463] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:49.432528 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9880e7d303b54d959bca4e8464a6b30c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.439116 1164 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.440589 1164 tablet_bootstrap.cc:654] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.447109 1185 ts_tablet_manager.cc:1918] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.456127 2563 consensus_queue.cc:1035] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.449632 847 leader_election.cc:304] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:49.459678 876 tablet_service.cc:1467] Processing CreateTablet for tablet c17e8808d62d4b7da88bffe7d7569568 (DEFAULT_TABLE table=default.Table_eba10319e22144a4bb6eee085519a0fb [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.463526 2975 raft_consensus.cc:2798] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.463668 2975 raft_consensus.cc:491] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.463840 2975 raft_consensus.cc:3054] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.463718 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17e8808d62d4b7da88bffe7d7569568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.333263 885 tablet_service.cc:1467] Processing CreateTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 (DEFAULT_TABLE table=default.Table_4a7a33e2dd7b42ada35718df1413bdd4 [id=70aa77642969417ca31d9b112034370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.467314 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096ee7d3a66846f9b619d8f2fc046616. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.466872 2975 raft_consensus.cc:513] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.470331 2975 leader_election.cc:290] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:49.472965 847 leader_election.cc:343] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.473249 847 leader_election.cc:343] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.474195 3082 mvcc.cc:204] Tried to move back new op lower bound from 7113585086217318400 to 7113585080732659712. Current Snapshot: MvccSnapshot[applied={T|T < 7113585086217318400}]
I20250112 20:51:49.474431 3097 tablet.cc:1721] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:49.479517 1012 tablet_service.cc:1467] Processing CreateTablet for tablet c17e8808d62d4b7da88bffe7d7569568 (DEFAULT_TABLE table=default.Table_eba10319e22144a4bb6eee085519a0fb [id=b5c55f1217cb4bada8cba93ada44f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.480137 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17e8808d62d4b7da88bffe7d7569568. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.491511 3097 tablet.cc:1721] T 161951bfabff485dbf7dceeac2d4128d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:49.498063 847 leader_election.cc:343] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.500013 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bea4d83ea644a0930a29f61e7a5fc8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.500460 1047 raft_consensus.cc:3054] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.501089 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c3812f8aa9ba422f85f331648a1ec02e (DEFAULT_TABLE table=default.Table_bdcb34e5580647dba61719814e3ce733 [id=3c0f9fd2807646e09df8875e8433f90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.501534 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3812f8aa9ba422f85f331648a1ec02e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.503695 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250112 20:51:49.506855 3086 tablet.cc:1721] T cc90d3722fd14d59b9994655cd30ef2b P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:49.503361 1047 raft_consensus.cc:2463] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:49.508514 847 leader_election.cc:304] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:49.508734 2521 raft_consensus.cc:2798] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.508884 2521 raft_consensus.cc:695] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:49.509124 2521 consensus_queue.cc:237] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.510339 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:49.510394 3085 tablet.cc:1721] T 161951bfabff485dbf7dceeac2d4128d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:49.515600 2447 ts_tablet_manager.cc:1918] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.515739 2447 log.cc:1198] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/cbc22207e9204b73877c74f61ebba8a8
I20250112 20:51:49.516078 2447 ts_tablet_manager.cc:1939] T cbc22207e9204b73877c74f61ebba8a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.516767 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 931) took 1970 ms. Trace:
I20250112 20:51:49.516871 2447 rpcz_store.cc:276] 0112 20:51:47.546698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.546746 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:49.163237 (+1616491us) ts_tablet_manager.cc:1145] Deleting tablet cbc22207e9204b73877c74f61ebba8a8
0112 20:51:49.163251 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.348957 (+185706us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.461345 (+112388us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.516749 (+ 55404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1616182}
I20250112 20:51:49.517148 2447 tablet_replica.cc:331] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:49.517308 2447 raft_consensus.cc:2238] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:49.462450 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8229e4ef771b4032a4bfae3190a851e3 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
W20250112 20:51:49.517767 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 55362.1us
W20250112 20:51:49.518020 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2de03e3114d145aeb81590dbae2c548e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de03e3114d145aeb81590dbae2c548e already in progress: creating tablet
I20250112 20:51:49.519119 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_594e1478361d4b56a02d64049420de56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.519691 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_594e1478361d4b56a02d64049420de56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.520915 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_53a9e79cf0434f96bdffc0b5c7f26934" } new_table_name: "default.Table_a6251ae914764a8f97588b9bb45053f2" modify_external_catalogs: true
I20250112 20:51:49.521159 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d] alter complete (version 1)
I20250112 20:51:49.522079 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae66395240e9403a94131921dfe99d18" } modify_external_catalogs: true
I20250112 20:51:49.522253 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ae66395240e9403a94131921dfe99d18" } modify_external_catalogs: true
I20250112 20:51:49.522486 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_36ebfe68c6e34958b6283665c113c466"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.523130 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36ebfe68c6e34958b6283665c113c466 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.523497 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_11813bf0d59d467d8f57da28e92d1b9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.523916 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11813bf0d59d467d8f57da28e92d1b9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.524386 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 7623bebb539e485aa508e95a35225979 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:49.524726 442 consensus_peers.cc:487] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05 -> Peer b985fd656ce44131bc91ccfeb79587c5 (127.0.104.61:39271): Couldn't send request to peer b985fd656ce44131bc91ccfeb79587c5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39271: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250112 20:51:49.525367 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bdcb34e5580647dba61719814e3ce733" } modify_external_catalogs: true
I20250112 20:51:49.525512 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bdcb34e5580647dba61719814e3ce733" } modify_external_catalogs: true
I20250112 20:51:49.526583 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_127c6f1785a7428793ecea2c36cf7a2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.526993 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_127c6f1785a7428793ecea2c36cf7a2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.527945 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_285cfdca273f457c98ac4d491dfe84af" } new_table_name: "default.Table_5147d452317249dd9a587356732a3d59" modify_external_catalogs: true
I20250112 20:51:49.529047 1185 log.cc:1198] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:49.529453 1185 ts_tablet_manager.cc:1939] T 8c5b10576bd34af5b02f8d8dcc3759ab P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:49.529580 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7009bf4316cd40bb9c4d7ea687dd7d03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.529963 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7009bf4316cd40bb9c4d7ea687dd7d03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.530368 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 973) took 1738 ms. Trace:
I20250112 20:51:49.530454 1185 rpcz_store.cc:276] 0112 20:51:47.791807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.791858 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:49.311555 (+1519697us) ts_tablet_manager.cc:1145] Deleting tablet 8c5b10576bd34af5b02f8d8dcc3759ab
0112 20:51:49.311568 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.444080 (+132512us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.447035 (+ 2955us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.530345 (+ 83310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519529}
I20250112 20:51:49.530786 1185 tablet_replica.cc:331] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:49.531006 1185 raft_consensus.cc:2238] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.531159 1185 raft_consensus.cc:2267] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.531755 1185 ts_tablet_manager.cc:1905] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:49.454938 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 98504.3us
I20250112 20:51:49.456743 752 tablet_service.cc:1467] Processing CreateTablet for tablet ced81345e31a4634bb21e3702ad69c93 (DEFAULT_TABLE table=default.Table_ae66395240e9403a94131921dfe99d18 [id=a7f0bc0c9fce44638273cfdcf3c68d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.533380 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced81345e31a4634bb21e3702ad69c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.538594 1163 tablet_bootstrap.cc:492] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.538745 1163 ts_tablet_manager.cc:1397] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.000s
I20250112 20:51:49.539664 1163 raft_consensus.cc:357] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.539848 1163 raft_consensus.cc:383] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.541424 3097 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.517916 1164 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.543781 1164 ts_tablet_manager.cc:1397] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.008s
I20250112 20:51:49.544701 1164 raft_consensus.cc:357] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.544952 1164 raft_consensus.cc:383] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.545028 1164 raft_consensus.cc:738] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.551225 2886 raft_consensus.cc:491] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.551363 2886 raft_consensus.cc:513] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.551885 2886 leader_election.cc:290] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.552000 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9880e7d303b54d959bca4e8464a6b30c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:49.552201 772 raft_consensus.cc:2463] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:49.552603 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bea4d83ea644a0930a29f61e7a5fc8" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:49.552759 772 raft_consensus.cc:3054] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.555198 772 raft_consensus.cc:2463] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:49.556221 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.556280 1164 consensus_queue.cc:260] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.556411 911 raft_consensus.cc:2463] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:49.556751 1164 ts_tablet_manager.cc:1428] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250112 20:51:49.557029 1164 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.558468 2632 consensus_queue.cc:1035] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.559269 710 leader_election.cc:304] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.559505 2632 raft_consensus.cc:2798] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.559571 2632 raft_consensus.cc:491] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.559624 2632 raft_consensus.cc:3054] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.562645 2632 raft_consensus.cc:513] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.563397 2632 leader_election.cc:290] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.569252 2968 consensus_queue.cc:1035] T fe7e150c031e4f4e99966c14e9005d00 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.569638 1164 tablet_bootstrap.cc:654] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:49.575789 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e5c228264e345348c9a1bf4357e7964 consensus state has no leader
I20250112 20:51:49.586306 1178 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:49.586503 1178 ts_tablet_manager.cc:1397] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.002s
I20250112 20:51:49.587323 1178 raft_consensus.cc:357] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.587544 1178 raft_consensus.cc:383] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.587319 2447 raft_consensus.cc:2267] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.587635 1178 raft_consensus.cc:738] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.587860 1178 consensus_queue.cc:260] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.588160 2447 ts_tablet_manager.cc:1905] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.588366 1178 ts_tablet_manager.cc:1428] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:49.588569 1186 log.cc:1198] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3dba3d9315db4b9292288ee62c2244bc
I20250112 20:51:49.589154 1186 ts_tablet_manager.cc:1939] T 3dba3d9315db4b9292288ee62c2244bc P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:49.593611 752 tablet_service.cc:1467] Processing CreateTablet for tablet ff43b277c7e64bbe887ec73b542ab47c (DEFAULT_TABLE table=default.Table_6302bcdd216e470db6dbad8bbcfee6f3 [id=1f63aeb199e045878d010c3abf01127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.595866 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff43b277c7e64bbe887ec73b542ab47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.539913 1163 raft_consensus.cc:738] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.601661 1163 consensus_queue.cc:260] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.602217 1163 ts_tablet_manager.cc:1428] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250112 20:51:49.602460 1163 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:49.602998 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 45584.4us
I20250112 20:51:49.603291 2983 consensus_queue.cc:1035] T cc90d3722fd14d59b9994655cd30ef2b P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:49.604279 847 leader_election.cc:343] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.604439 847 leader_election.cc:304] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:49.604691 2983 raft_consensus.cc:2743] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:49.606715 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 946) took 3089 ms. Trace:
I20250112 20:51:49.606814 1186 rpcz_store.cc:276] 0112 20:51:46.517592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.517637 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:49.420238 (+2902601us) ts_tablet_manager.cc:1145] Deleting tablet 3dba3d9315db4b9292288ee62c2244bc
0112 20:51:49.420249 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.424695 (+ 4446us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.427802 (+ 3107us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.606675 (+178873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902044}
I20250112 20:51:49.607125 1186 tablet_replica.cc:331] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:49.607383 847 leader_election.cc:304] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:49.607693 2975 raft_consensus.cc:2798] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.607820 2975 raft_consensus.cc:695] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:49.608155 2975 consensus_queue.cc:237] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.609953 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:49.611686 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_93126de589c046d09407fb32545376cc" } modify_external_catalogs: true
I20250112 20:51:49.611860 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_93126de589c046d09407fb32545376cc" } modify_external_catalogs: true
I20250112 20:51:49.612174 847 leader_election.cc:304] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:49.617136 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ced81345e31a4634bb21e3702ad69c93
I20250112 20:51:49.617257 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.617437 906 raft_consensus.cc:3054] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.619439 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 7623bebb539e485aa508e95a35225979
I20250112 20:51:49.619789 452 catalog_manager.cc:6596] ReplaceTablet: tablet 7623bebb539e485aa508e95a35225979 deleted and replaced by tablet 9eb4ed0e18e7444c9173626df8b305b8
I20250112 20:51:49.621328 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c3812f8aa9ba422f85f331648a1ec02e
I20250112 20:51:49.621359 752 tablet_service.cc:1467] Processing CreateTablet for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb (DEFAULT_TABLE table=default.Table_c57b973f9ff045d687f8b5609c1941fa [id=50171ecbdde5455da7ac3f8a2ab491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.621718 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.621963 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f6995f7fbbf480995353b66749fbc10 from {username='slave'} at 127.0.0.1:55686
W20250112 20:51:49.621994 710 consensus_peers.cc:487] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 669e1cfd407e4d87a63e6832d9c6fa1f. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:49.625140 3085 tablet.cc:1721] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.625248 744 tablet_service.cc:1467] Processing CreateTablet for tablet 648ec1632b404cf0a21c0cfc0b2711c5 (DEFAULT_TABLE table=default.Table_662121681f354c6a9f08a434efbafcce [id=ed5f608218414eb992f1801d0f5d7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.625555 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ec1632b404cf0a21c0cfc0b2711c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.626343 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_95f1ea8dfc6a4580a7b027fcc460ca76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.626818 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95f1ea8dfc6a4580a7b027fcc460ca76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.628329 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_963ba961a67b40e08a07744599f1673d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.628803 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_963ba961a67b40e08a07744599f1673d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.629930 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_737195573b924b53b18c84b5b0f7f44c" } new_table_name: "default.Table_7f87f7dbd08e4a5abb17ed757bbe20de" modify_external_catalogs: true
I20250112 20:51:49.631230 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1de9e41db6aa48deb9875ff6f40a87bf" } modify_external_catalogs: true
I20250112 20:51:49.631395 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1de9e41db6aa48deb9875ff6f40a87bf" } modify_external_catalogs: true
I20250112 20:51:49.632263 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.632521 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eba10319e22144a4bb6eee085519a0fb" } new_table_name: "default.Table_ebc936227805419b91c80e6d8a21e144" modify_external_catalogs: true
I20250112 20:51:49.633052 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4147bd0534264bf99481a95b8e5120ff (DEFAULT_TABLE table=default.Table_04069d4b7bad4757988c3e7e8ab02b44 [id=a05df3d4e76244708d3d8da90fa6e428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.633301 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.633395 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4147bd0534264bf99481a95b8e5120ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.634337 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1c9cc0e4c5ea4333a8e2fcc16ff4ac0f" } new_table_name: "default.Table_f5df1b9a2b9947849b4948b8dcdfdc09" modify_external_catalogs: true
I20250112 20:51:49.637427 753 tablet_service.cc:1467] Processing CreateTablet for tablet 89e2d166a2a24e8ca4830936171dd36d (DEFAULT_TABLE table=default.Table_2f77351ffe3d4bb995bee57e885a76de [id=6071e3cfbb2147259555da613232515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.637775 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2d166a2a24e8ca4830936171dd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.639586 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 648ec1632b404cf0a21c0cfc0b2711c5 (DEFAULT_TABLE table=default.Table_662121681f354c6a9f08a434efbafcce [id=ed5f608218414eb992f1801d0f5d7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.639945 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ec1632b404cf0a21c0cfc0b2711c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.641104 738 tablet_service.cc:1467] Processing CreateTablet for tablet d4af0222bd74408abbaddc71d9ebca24 (DEFAULT_TABLE table=default.Table_46854461640b4fec8b578900ba36d90e [id=34097b8105104c6394b9b7d5515b4ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.641420 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0222bd74408abbaddc71d9ebca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.645092 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb (DEFAULT_TABLE table=default.Table_c57b973f9ff045d687f8b5609c1941fa [id=50171ecbdde5455da7ac3f8a2ab491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.645428 3122 raft_consensus.cc:491] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.645576 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.645532 3122 raft_consensus.cc:513] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.646114 1163 tablet_bootstrap.cc:654] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.646137 3122 leader_election.cc:290] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.647393 747 tablet_service.cc:1467] Processing CreateTablet for tablet d99f1ae971034e87a92136cc812aa13b (DEFAULT_TABLE table=default.Table_e45ad07d53b94b6f803b6b694656de77 [id=bce23ca5685c455e8607ba38884cf507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.647711 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d99f1ae971034e87a92136cc812aa13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.648890 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.649243 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 4147bd0534264bf99481a95b8e5120ff (DEFAULT_TABLE table=default.Table_04069d4b7bad4757988c3e7e8ab02b44 [id=a05df3d4e76244708d3d8da90fa6e428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.649529 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4147bd0534264bf99481a95b8e5120ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.653658 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4adcd8abec459d8cb1e34201a0575a (DEFAULT_TABLE table=default.Table_62da44136d7d4f6dbc73e917cf24ad6f [id=fc1e3c37a7a047be98e76606a09a191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.653992 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c4adcd8abec459d8cb1e34201a0575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.654273 750 tablet_service.cc:1467] Processing CreateTablet for tablet e2be779fac1e4240a943e7862419bb82 (DEFAULT_TABLE table=default.Table_799f579511f14383a50065deef83f536 [id=08904694214d4d34b41ef3ff117f9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.654592 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2be779fac1e4240a943e7862419bb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.657970 1010 tablet_service.cc:1467] Processing CreateTablet for tablet ff43b277c7e64bbe887ec73b542ab47c (DEFAULT_TABLE table=default.Table_6302bcdd216e470db6dbad8bbcfee6f3 [id=1f63aeb199e045878d010c3abf01127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.658354 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff43b277c7e64bbe887ec73b542ab47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.658318 746 tablet_service.cc:1467] Processing CreateTablet for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf (DEFAULT_TABLE table=default.Table_a51f9960ef424b2e9ccca224a6a1654e [id=491e07b78e4347de82b26f36ba8abcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.658601 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.661021 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8683da60f47f48a1a82b85f0193bf1a2 (DEFAULT_TABLE table=default.Table_040d82aa49ea41d8b4bdfe798a1ae64c [id=4a8258a73fe04f1cb4a923fc1cdaa2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.661309 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8683da60f47f48a1a82b85f0193bf1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.663380 1164 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.663492 1164 ts_tablet_manager.cc:1397] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.000s
I20250112 20:51:49.664304 739 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5e76413fc44419335a6c953013e41 (DEFAULT_TABLE table=default.Table_691a39c63d3a4966914ce5e983c87d47 [id=70a538ff208a4a7ca7676c1181b9cc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.664613 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5e76413fc44419335a6c953013e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.664615 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f67bc41b96f8402bbefe2a5a5be8ffd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.665180 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f67bc41b96f8402bbefe2a5a5be8ffd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.665163 1016 tablet_service.cc:1467] Processing CreateTablet for tablet d99f1ae971034e87a92136cc812aa13b (DEFAULT_TABLE table=default.Table_e45ad07d53b94b6f803b6b694656de77 [id=bce23ca5685c455e8607ba38884cf507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.665511 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d99f1ae971034e87a92136cc812aa13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.667768 2447 ts_tablet_manager.cc:1918] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.667687 754 tablet_service.cc:1467] Processing CreateTablet for tablet 881bb8d31e59475e809c9726e4bf6716 (DEFAULT_TABLE table=default.Table_1f568569d7234222a4ea47ebca7c2170 [id=d9ae65be13e04b2086b2ac434f74e563]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.667869 2447 log.cc:1198] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/577dc789f0cb48f49168124b37a6fba0
I20250112 20:51:49.668093 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881bb8d31e59475e809c9726e4bf6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.668227 2447 ts_tablet_manager.cc:1939] T 577dc789f0cb48f49168124b37a6fba0 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.669095 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4adcd8abec459d8cb1e34201a0575a (DEFAULT_TABLE table=default.Table_62da44136d7d4f6dbc73e917cf24ad6f [id=fc1e3c37a7a047be98e76606a09a191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.669404 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c4adcd8abec459d8cb1e34201a0575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.669582 737 tablet_service.cc:1467] Processing CreateTablet for tablet 30321942214949b3bcb727d49f4a34d6 (DEFAULT_TABLE table=default.Table_72b8c68194b84004b32fa74e32d5e26e [id=38e81f9757ab481f99dbd49ab35e534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.669842 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30321942214949b3bcb727d49f4a34d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.670519 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf (DEFAULT_TABLE table=default.Table_a51f9960ef424b2e9ccca224a6a1654e [id=491e07b78e4347de82b26f36ba8abcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.670815 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.671061 741 tablet_service.cc:1467] Processing CreateTablet for tablet 407a01e5f1524bc48ddc5d6c7c1e8294 (DEFAULT_TABLE table=default.Table_7e596a69a2ad4a0abbc10f0b7f9686fd [id=d56867b070ee488599d1c7098e79abee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.671334 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407a01e5f1524bc48ddc5d6c7c1e8294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.674403 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e2be779fac1e4240a943e7862419bb82 (DEFAULT_TABLE table=default.Table_799f579511f14383a50065deef83f536 [id=08904694214d4d34b41ef3ff117f9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.674686 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2be779fac1e4240a943e7862419bb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.674855 1164 raft_consensus.cc:357] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.675040 1164 raft_consensus.cc:383] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.675114 1164 raft_consensus.cc:738] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.675292 1164 consensus_queue.cc:260] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.675782 1164 ts_tablet_manager.cc:1428] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250112 20:51:49.676046 1164 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.676515 1163 tablet_bootstrap.cc:492] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.676628 1163 ts_tablet_manager.cc:1397] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.001s
I20250112 20:51:49.677071 756 tablet_service.cc:1467] Processing CreateTablet for tablet cef28afa25744cc785fc68592f02ed0e (DEFAULT_TABLE table=default.Table_1e9b4851670045b1975a05cdd3f5c2f0 [id=e63f154df1304ae39f1c1b462bf1515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.677371 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef28afa25744cc785fc68592f02ed0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.677333 1163 raft_consensus.cc:357] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.677484 1163 raft_consensus.cc:383] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.677524 1163 raft_consensus.cc:738] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.677675 1163 consensus_queue.cc:260] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.678654 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8683da60f47f48a1a82b85f0193bf1a2 (DEFAULT_TABLE table=default.Table_040d82aa49ea41d8b4bdfe798a1ae64c [id=4a8258a73fe04f1cb4a923fc1cdaa2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.678920 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8683da60f47f48a1a82b85f0193bf1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.679740 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d4af0222bd74408abbaddc71d9ebca24 (DEFAULT_TABLE table=default.Table_46854461640b4fec8b578900ba36d90e [id=34097b8105104c6394b9b7d5515b4ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.680011 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0222bd74408abbaddc71d9ebca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.680305 749 tablet_service.cc:1467] Processing CreateTablet for tablet de498568093d4e1a9f91d3d5349bb7fa (DEFAULT_TABLE table=default.Table_3fae1e5ce26f4ad593ad310c09385e88 [id=5517a0ca3e504993b1b7febb8d41dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.680629 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de498568093d4e1a9f91d3d5349bb7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.685390 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5e76413fc44419335a6c953013e41 (DEFAULT_TABLE table=default.Table_691a39c63d3a4966914ce5e983c87d47 [id=70a538ff208a4a7ca7676c1181b9cc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.685989 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5e76413fc44419335a6c953013e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.686640 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 881bb8d31e59475e809c9726e4bf6716 (DEFAULT_TABLE table=default.Table_1f568569d7234222a4ea47ebca7c2170 [id=d9ae65be13e04b2086b2ac434f74e563]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.686981 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881bb8d31e59475e809c9726e4bf6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.691756 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 30321942214949b3bcb727d49f4a34d6 (DEFAULT_TABLE table=default.Table_72b8c68194b84004b32fa74e32d5e26e [id=38e81f9757ab481f99dbd49ab35e534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.692124 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30321942214949b3bcb727d49f4a34d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.692621 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6f89ff69e849b4abbca69a4a6bc638 (DEFAULT_TABLE table=default.Table_86e4035224cc49598a669f49ad4553e4 [id=89b8231828d4456abdc8a1dc5ca22e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.692997 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6f89ff69e849b4abbca69a4a6bc638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.693317 755 tablet_service.cc:1467] Processing CreateTablet for tablet 544ea9d983794cf69ec68d96ec5e0438 (DEFAULT_TABLE table=default.Table_6188282bedce4235988a0602b6addcaa [id=6c1e08f27aff4cfb8b3b09eeda4c3d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.693614 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544ea9d983794cf69ec68d96ec5e0438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.697019 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 407a01e5f1524bc48ddc5d6c7c1e8294 (DEFAULT_TABLE table=default.Table_7e596a69a2ad4a0abbc10f0b7f9686fd [id=d56867b070ee488599d1c7098e79abee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.697242 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_c8b8e3e3e6234cfd952aeb5116a142f9 [id=fdfa5636686548deb94302c32d7a1757] alter complete (version 4)
I20250112 20:51:49.697522 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407a01e5f1524bc48ddc5d6c7c1e8294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.700635 755 tablet_service.cc:1467] Processing CreateTablet for tablet 752cd0d20fef49ad9bcfc0d3cbec8430 (DEFAULT_TABLE table=default.Table_737195573b924b53b18c84b5b0f7f44c [id=3252a9a2ddb440d5a7ee6236d4a95e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.701058 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752cd0d20fef49ad9bcfc0d3cbec8430. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.701026 1022 tablet_service.cc:1467] Processing CreateTablet for tablet cef28afa25744cc785fc68592f02ed0e (DEFAULT_TABLE table=default.Table_1e9b4851670045b1975a05cdd3f5c2f0 [id=e63f154df1304ae39f1c1b462bf1515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.701354 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef28afa25744cc785fc68592f02ed0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.701750 2632 raft_consensus.cc:491] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.701925 2632 raft_consensus.cc:513] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.702507 2632 leader_election.cc:290] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.706451 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c90c8dd4dfb14507accb82828be512f0
I20250112 20:51:49.708498 755 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4bd27cc30c422d85470c5a2df5d3ab (DEFAULT_TABLE table=default.Table_6763d75a41e34bc7aee13341771b4c7e [id=e94da089ebf9424bbd5017d1a4da65c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.708878 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4bd27cc30c422d85470c5a2df5d3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.711191 1164 tablet_bootstrap.cc:654] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.711989 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 752cd0d20fef49ad9bcfc0d3cbec8430 (DEFAULT_TABLE table=default.Table_737195573b924b53b18c84b5b0f7f44c [id=3252a9a2ddb440d5a7ee6236d4a95e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.712400 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752cd0d20fef49ad9bcfc0d3cbec8430. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.713596 1163 ts_tablet_manager.cc:1428] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250112 20:51:49.713848 1163 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:49.715459 1163 tablet_bootstrap.cc:654] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.607384 1186 raft_consensus.cc:2238] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.716547 740 tablet_service.cc:1467] Processing CreateTablet for tablet c90c8dd4dfb14507accb82828be512f0 (DEFAULT_TABLE table=default.Table_93126de589c046d09407fb32545376cc [id=9d7d96bc6fd04f0c99acd0011810323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.716877 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90c8dd4dfb14507accb82828be512f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.717011 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8923b3ba7b44d080c19cd0e2103c49 (DEFAULT_TABLE table=default.Table_7e689916a41242d4bc7eec11c33be6b5 [id=c648a25ab5b14a25a7b1ed5c10ba4f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.717316 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8923b3ba7b44d080c19cd0e2103c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.721630 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c90c8dd4dfb14507accb82828be512f0 (DEFAULT_TABLE table=default.Table_93126de589c046d09407fb32545376cc [id=9d7d96bc6fd04f0c99acd0011810323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.721943 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90c8dd4dfb14507accb82828be512f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.724064 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 89e2d166a2a24e8ca4830936171dd36d (DEFAULT_TABLE table=default.Table_2f77351ffe3d4bb995bee57e885a76de [id=6071e3cfbb2147259555da613232515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.724138 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8fb96959a4485b6f7e76a89830681 (DEFAULT_TABLE table=default.Table_162669e16db04dd4a6ea5aa5fd553e4f [id=0387b248aa5745be828d7da2620d6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.724349 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2d166a2a24e8ca4830936171dd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.724529 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a8fb96959a4485b6f7e76a89830681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.729351 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 933) took 2182 ms. Trace:
I20250112 20:51:49.730455 2447 rpcz_store.cc:276] 0112 20:51:47.547161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.547204 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:49.517099 (+1969895us) ts_tablet_manager.cc:1145] Deleting tablet 577dc789f0cb48f49168124b37a6fba0
0112 20:51:49.517107 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.636493 (+119386us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.667691 (+ 31198us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.729331 (+ 61640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969775}
I20250112 20:51:49.731823 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet cbc22207e9204b73877c74f61ebba8a8 (table default.Table_16d626d49adf42eea0a98976db7a45b0 [id=3ab1129e25204a9fa1603306235c6ac5]) successfully deleted
I20250112 20:51:49.733394 3122 consensus_queue.cc:1035] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.730993 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.733747 3142 consensus_queue.cc:1035] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.732568 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549d92191a594788bf639cd60a6b5142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:49.734126 2447 tablet_replica.cc:331] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:49.734334 2447 raft_consensus.cc:2238] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:49.734465 710 leader_election.cc:343] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.734799 710 leader_election.cc:343] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.735611 2787 consensus_queue.cc:1035] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.741053 2447 raft_consensus.cc:2267] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.741962 2447 ts_tablet_manager.cc:1905] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.747224 2521 raft_consensus.cc:2798] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.747411 1186 raft_consensus.cc:2267] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.748317 1178 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:49.749250 750 tablet_service.cc:1467] Processing CreateTablet for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3 (DEFAULT_TABLE table=default.Table_4dc07bdc6b32422eb695a7f4f5710fce [id=4bb32a478b22437c913b2e02e309b6f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:49.750325 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 134635us
I20250112 20:51:49.750526 2521 raft_consensus.cc:491] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.747869 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6f89ff69e849b4abbca69a4a6bc638 (DEFAULT_TABLE table=default.Table_86e4035224cc49598a669f49ad4553e4 [id=89b8231828d4456abdc8a1dc5ca22e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.751312 1186 ts_tablet_manager.cc:1905] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.751585 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.751345 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6f89ff69e849b4abbca69a4a6bc638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.748721 1163 tablet_bootstrap.cc:492] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.752974 1163 ts_tablet_manager.cc:1397] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250112 20:51:49.753684 1163 raft_consensus.cc:357] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.754544 2521 raft_consensus.cc:3054] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.756948 1163 raft_consensus.cc:383] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.757035 1163 raft_consensus.cc:738] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.757241 1163 consensus_queue.cc:260] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.757905 1163 ts_tablet_manager.cc:1428] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:49.758168 1163 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:49.758371 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 752cd0d20fef49ad9bcfc0d3cbec8430 consensus state has no leader
I20250112 20:51:49.758335 2521 raft_consensus.cc:513] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.758910 2521 leader_election.cc:290] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.759487 1163 tablet_bootstrap.cc:654] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.761080 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8c5b10576bd34af5b02f8d8dcc3759ab (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:49.761300 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 89f9b46da06a476ab9f2a93ca74ff736 (table default.Table_a6c9111b483148718ccd2bfb2a54698b [id=b3b49691fa8a41dda2abad8d2f59e423]) successfully deleted
W20250112 20:51:49.761541 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 015f54bdae6b4333963f9e085c5189e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015f54bdae6b4333963f9e085c5189e8 already in progress: creating tablet
W20250112 20:51:49.761863 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 0d072ebae78e40c48e158eb65775476c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d072ebae78e40c48e158eb65775476c already in progress: creating tablet
W20250112 20:51:49.762212 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3f6ad2ac50c745448031bdf94e0eea59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f6ad2ac50c745448031bdf94e0eea59 already in progress: creating tablet
W20250112 20:51:49.762557 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 69012f97aac1441095aa3bb1548de9cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69012f97aac1441095aa3bb1548de9cd already in progress: creating tablet
I20250112 20:51:49.762907 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet c2664a2e98f74f73ab2dc0adaa2335d3 (table default.Table_42c5efdd34e643cba03f44dc9c276e53 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]) successfully deleted
W20250112 20:51:49.763113 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b0786594542416cb67c4112921943b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0786594542416cb67c4112921943b3 already in progress: creating tablet
I20250112 20:51:49.763305 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 01367a11dea04b33a9255508501b9060 (table default.Table_cad323ed70af4a0a91136d819296a4da [id=b38280bc19954347952094e834eab557]) successfully deleted
W20250112 20:51:49.763504 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet fc1335522a384eb98163a2539630b93e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1335522a384eb98163a2539630b93e already in progress: creating tablet
I20250112 20:51:49.763612 2787 consensus_queue.cc:1035] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:49.763819 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet be1f12369c894682b7d30de4e37bd466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1f12369c894682b7d30de4e37bd466 already in progress: creating tablet
W20250112 20:51:49.764030 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a10f7a47089a43d8ae9c2fc16d379740 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10f7a47089a43d8ae9c2fc16d379740 already in progress: creating tablet
I20250112 20:51:49.764017 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8923b3ba7b44d080c19cd0e2103c49 (DEFAULT_TABLE table=default.Table_7e689916a41242d4bc7eec11c33be6b5 [id=c648a25ab5b14a25a7b1ed5c10ba4f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.764222 1178 tablet_bootstrap.cc:654] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:49.764279 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2aef567a7cf64178b53625ac4638048b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aef567a7cf64178b53625ac4638048b already in progress: creating tablet
W20250112 20:51:49.764468 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a2dbdbf416f64011b431f5188e0a06c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbdbf416f64011b431f5188e0a06c4 already in progress: creating tablet
I20250112 20:51:49.764493 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8923b3ba7b44d080c19cd0e2103c49. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.764678 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 53518b453d224ca186fcc6b2ffab5f04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53518b453d224ca186fcc6b2ffab5f04 already in progress: creating tablet
W20250112 20:51:49.764900 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 734734ff9c564f769bea0984a9e0386b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 734734ff9c564f769bea0984a9e0386b already in progress: creating tablet
W20250112 20:51:49.765125 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e92d58058511437d830a8f3cbba1e206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e92d58058511437d830a8f3cbba1e206 already in progress: creating tablet
I20250112 20:51:49.765197 1014 tablet_service.cc:1467] Processing CreateTablet for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3 (DEFAULT_TABLE table=default.Table_4dc07bdc6b32422eb695a7f4f5710fce [id=4bb32a478b22437c913b2e02e309b6f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:49.765336 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c6bdfecd5d9142d5b008c613864c8217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bdfecd5d9142d5b008c613864c8217 already in progress: creating tablet
I20250112 20:51:49.765538 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.765586 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f0fb6a5146c44edea076a10da58ea1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0fb6a5146c44edea076a10da58ea1a7 already in progress: creating tablet
W20250112 20:51:49.765800 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f424288fe62849edbbd2d38e89331fb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f424288fe62849edbbd2d38e89331fb7 already in progress: creating tablet
I20250112 20:51:49.766026 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet c8d24bccf682483aa9b273fc1ce847da (table default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]) successfully deleted
I20250112 20:51:49.766216 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 669e1cfd407e4d87a63e6832d9c6fa1f (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
I20250112 20:51:49.766307 741 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8fb96959a4485b6f7e76a89830681 (DEFAULT_TABLE table=default.Table_162669e16db04dd4a6ea5aa5fd553e4f [id=0387b248aa5745be828d7da2620d6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.766507 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 7f807071e53a4f0e8629b1dfa6d26b9d (table default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]) successfully deleted
I20250112 20:51:49.766600 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a8fb96959a4485b6f7e76a89830681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.766690 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 924082716c1e4d9ea6fbe97ae93232bd (table default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]) successfully deleted
I20250112 20:51:49.767350 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet dc3a9740415140fe8864d54f477af954 (table default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]) successfully deleted
I20250112 20:51:49.767665 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9880e7d303b54d959bca4e8464a6b30c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:49.767858 1047 raft_consensus.cc:3054] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.770761 1047 raft_consensus.cc:2463] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:49.778993 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.779186 905 raft_consensus.cc:2463] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:49.780393 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549d92191a594788bf639cd60a6b5142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:49.780573 905 raft_consensus.cc:2463] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:49.781924 2447 ts_tablet_manager.cc:1918] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:49.782029 2447 log.cc:1198] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/2e6b4882a3d744bda5677aa9e01a53f4
I20250112 20:51:49.782339 2447 ts_tablet_manager.cc:1939] T 2e6b4882a3d744bda5677aa9e01a53f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.783090 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 934) took 2235 ms. Trace:
I20250112 20:51:49.783203 2447 rpcz_store.cc:276] 0112 20:51:47.547256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.547300 (+ 44us) service_pool.cc:225] Handling call
0112 20:51:49.734082 (+2186782us) ts_tablet_manager.cc:1145] Deleting tablet 2e6b4882a3d744bda5677aa9e01a53f4
0112 20:51:49.734091 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.779001 (+ 44910us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.781854 (+ 2853us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.783072 (+ 1218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2186636}
I20250112 20:51:49.783569 2447 tablet_replica.cc:331] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:49.783747 2447 raft_consensus.cc:2238] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.783895 2447 raft_consensus.cc:2267] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.784369 2447 ts_tablet_manager.cc:1905] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.787417 906 raft_consensus.cc:2463] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:49.795792 1163 tablet_bootstrap.cc:492] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.795928 1163 ts_tablet_manager.cc:1397] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20250112 20:51:49.796626 1163 raft_consensus.cc:357] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.796823 1163 raft_consensus.cc:383] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.796885 1163 raft_consensus.cc:738] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.797087 1163 consensus_queue.cc:260] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.797595 1163 ts_tablet_manager.cc:1428] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:49.797874 1163 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:49.798182 1022 tablet_service.cc:1467] Processing CreateTablet for tablet de498568093d4e1a9f91d3d5349bb7fa (DEFAULT_TABLE table=default.Table_3fae1e5ce26f4ad593ad310c09385e88 [id=5517a0ca3e504993b1b7febb8d41dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.798909 876 tablet_service.cc:1467] Processing CreateTablet for tablet ff43b277c7e64bbe887ec73b542ab47c (DEFAULT_TABLE table=default.Table_6302bcdd216e470db6dbad8bbcfee6f3 [id=1f63aeb199e045878d010c3abf01127e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.799309 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff43b277c7e64bbe887ec73b542ab47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.799491 888 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4adcd8abec459d8cb1e34201a0575a (DEFAULT_TABLE table=default.Table_62da44136d7d4f6dbc73e917cf24ad6f [id=fc1e3c37a7a047be98e76606a09a191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.799849 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c4adcd8abec459d8cb1e34201a0575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.801142 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de498568093d4e1a9f91d3d5349bb7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.803321 885 tablet_service.cc:1467] Processing CreateTablet for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb (DEFAULT_TABLE table=default.Table_c57b973f9ff045d687f8b5609c1941fa [id=50171ecbdde5455da7ac3f8a2ab491b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.803686 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34feddfdeafe4eae8988e6ccdf2b9bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.800575 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4bd27cc30c422d85470c5a2df5d3ab (DEFAULT_TABLE table=default.Table_6763d75a41e34bc7aee13341771b4c7e [id=e94da089ebf9424bbd5017d1a4da65c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.804600 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4bd27cc30c422d85470c5a2df5d3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.805069 887 tablet_service.cc:1467] Processing CreateTablet for tablet d99f1ae971034e87a92136cc812aa13b (DEFAULT_TABLE table=default.Table_e45ad07d53b94b6f803b6b694656de77 [id=bce23ca5685c455e8607ba38884cf507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.800712 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 544ea9d983794cf69ec68d96ec5e0438 (DEFAULT_TABLE table=default.Table_6188282bedce4235988a0602b6addcaa [id=6c1e08f27aff4cfb8b3b09eeda4c3d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.805404 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d99f1ae971034e87a92136cc812aa13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.805433 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544ea9d983794cf69ec68d96ec5e0438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.809031 879 tablet_service.cc:1467] Processing CreateTablet for tablet 648ec1632b404cf0a21c0cfc0b2711c5 (DEFAULT_TABLE table=default.Table_662121681f354c6a9f08a434efbafcce [id=ed5f608218414eb992f1801d0f5d7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.809401 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ec1632b404cf0a21c0cfc0b2711c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.811429 710 leader_election.cc:304] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.811780 710 leader_election.cc:304] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:49.812114 880 tablet_service.cc:1467] Processing CreateTablet for tablet 30321942214949b3bcb727d49f4a34d6 (DEFAULT_TABLE table=default.Table_72b8c68194b84004b32fa74e32d5e26e [id=38e81f9757ab481f99dbd49ab35e534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.812450 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30321942214949b3bcb727d49f4a34d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.813094 875 tablet_service.cc:1467] Processing CreateTablet for tablet 4147bd0534264bf99481a95b8e5120ff (DEFAULT_TABLE table=default.Table_04069d4b7bad4757988c3e7e8ab02b44 [id=a05df3d4e76244708d3d8da90fa6e428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.813519 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4147bd0534264bf99481a95b8e5120ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.817029 890 tablet_service.cc:1467] Processing CreateTablet for tablet 89e2d166a2a24e8ca4830936171dd36d (DEFAULT_TABLE table=default.Table_2f77351ffe3d4bb995bee57e885a76de [id=6071e3cfbb2147259555da613232515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.817342 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e2d166a2a24e8ca4830936171dd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.817978 1185 ts_tablet_manager.cc:1918] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.818089 1185 log.cc:1198] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/577dc789f0cb48f49168124b37a6fba0
I20250112 20:51:49.818424 1185 ts_tablet_manager.cc:1939] T 577dc789f0cb48f49168124b37a6fba0 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:49.819408 1164 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.819695 1164 ts_tablet_manager.cc:1397] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.004s
I20250112 20:51:49.821089 889 tablet_service.cc:1467] Processing CreateTablet for tablet d4af0222bd74408abbaddc71d9ebca24 (DEFAULT_TABLE table=default.Table_46854461640b4fec8b578900ba36d90e [id=34097b8105104c6394b9b7d5515b4ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.821435 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4af0222bd74408abbaddc71d9ebca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.772140 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 961a72a9d8a643cea743f3f021bd5a03
W20250112 20:51:49.822749 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 55402.5us
W20250112 20:51:49.788645 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c17e8808d62d4b7da88bffe7d7569568 consensus state has no leader
W20250112 20:51:49.797976 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 consensus state has no leader
I20250112 20:51:49.815155 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]). Expected version 1 got 0
I20250112 20:51:49.823421 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:49.823695 450 catalog_manager.cc:5526] T 1b1565c7487c428f8f963aca406263fe P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:49.824090 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:49.824482 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 8c5b10576bd34af5b02f8d8dcc3759ab (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:49.824338 450 catalog_manager.cc:5526] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:49.824725 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1e584b761d234c5aa0ab875acd8e1e05 (table default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a]). Expected version 4 got 0
I20250112 20:51:49.825039 884 tablet_service.cc:1467] Processing CreateTablet for tablet e2be779fac1e4240a943e7862419bb82 (DEFAULT_TABLE table=default.Table_799f579511f14383a50065deef83f536 [id=08904694214d4d34b41ef3ff117f9523]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.825196 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:49.825374 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2be779fac1e4240a943e7862419bb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.825381 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:49.825729 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 161951bfabff485dbf7dceeac2d4128d (table default.Table_023f7c8ba47d421fbb789a60f7830d0b [id=60b6e28026a74197a4ba790310e1680d]). Expected version 1 got 0
I20250112 20:51:49.826133 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:49.829022 892 tablet_service.cc:1467] Processing CreateTablet for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf (DEFAULT_TABLE table=default.Table_a51f9960ef424b2e9ccca224a6a1654e [id=491e07b78e4347de82b26f36ba8abcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.829347 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e1f7c9d32341d0b9ea3ec63bd15daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.829457 1163 tablet_bootstrap.cc:654] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.833389 877 tablet_service.cc:1467] Processing CreateTablet for tablet 8683da60f47f48a1a82b85f0193bf1a2 (DEFAULT_TABLE table=default.Table_040d82aa49ea41d8b4bdfe798a1ae64c [id=4a8258a73fe04f1cb4a923fc1cdaa2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.833741 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8683da60f47f48a1a82b85f0193bf1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.837023 881 tablet_service.cc:1467] Processing CreateTablet for tablet 881bb8d31e59475e809c9726e4bf6716 (DEFAULT_TABLE table=default.Table_1f568569d7234222a4ea47ebca7c2170 [id=d9ae65be13e04b2086b2ac434f74e563]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.837355 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881bb8d31e59475e809c9726e4bf6716. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.841041 874 tablet_service.cc:1467] Processing CreateTablet for tablet 82f5e76413fc44419335a6c953013e41 (DEFAULT_TABLE table=default.Table_691a39c63d3a4966914ce5e983c87d47 [id=70a538ff208a4a7ca7676c1181b9cc5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.841385 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f5e76413fc44419335a6c953013e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.841686 1163 tablet_bootstrap.cc:492] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.841802 1163 ts_tablet_manager.cc:1397] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.002s
I20250112 20:51:49.842432 1163 raft_consensus.cc:357] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.842603 1163 raft_consensus.cc:383] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.842644 1163 raft_consensus.cc:738] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.843053 1163 consensus_queue.cc:260] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.843852 893 tablet_service.cc:1467] Processing CreateTablet for tablet de498568093d4e1a9f91d3d5349bb7fa (DEFAULT_TABLE table=default.Table_3fae1e5ce26f4ad593ad310c09385e88 [id=5517a0ca3e504993b1b7febb8d41dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.844197 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de498568093d4e1a9f91d3d5349bb7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.845005 878 tablet_service.cc:1467] Processing CreateTablet for tablet 407a01e5f1524bc48ddc5d6c7c1e8294 (DEFAULT_TABLE table=default.Table_7e596a69a2ad4a0abbc10f0b7f9686fd [id=d56867b070ee488599d1c7098e79abee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.845355 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407a01e5f1524bc48ddc5d6c7c1e8294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.849033 883 tablet_service.cc:1467] Processing CreateTablet for tablet cef28afa25744cc785fc68592f02ed0e (DEFAULT_TABLE table=default.Table_1e9b4851670045b1975a05cdd3f5c2f0 [id=e63f154df1304ae39f1c1b462bf1515d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.849385 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef28afa25744cc785fc68592f02ed0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.849943 1163 ts_tablet_manager.cc:1428] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:49.850198 1163 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:49.851495 1163 tablet_bootstrap.cc:654] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.851974 876 tablet_service.cc:1467] Processing CreateTablet for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3 (DEFAULT_TABLE table=default.Table_4dc07bdc6b32422eb695a7f4f5710fce [id=4bb32a478b22437c913b2e02e309b6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.852315 886 tablet_service.cc:1467] Processing CreateTablet for tablet 76a8fb96959a4485b6f7e76a89830681 (DEFAULT_TABLE table=default.Table_162669e16db04dd4a6ea5aa5fd553e4f [id=0387b248aa5745be828d7da2620d6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.852303 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2c96ab5de243f0a6adc8bc3b6c31d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.852803 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a8fb96959a4485b6f7e76a89830681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.853405 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6f89ff69e849b4abbca69a4a6bc638 (DEFAULT_TABLE table=default.Table_86e4035224cc49598a669f49ad4553e4 [id=89b8231828d4456abdc8a1dc5ca22e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.853688 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6f89ff69e849b4abbca69a4a6bc638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.856128 882 tablet_service.cc:1467] Processing CreateTablet for tablet c90c8dd4dfb14507accb82828be512f0 (DEFAULT_TABLE table=default.Table_93126de589c046d09407fb32545376cc [id=9d7d96bc6fd04f0c99acd0011810323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.856515 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90c8dd4dfb14507accb82828be512f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.857452 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
W20250112 20:51:49.857571 983 consensus_peers.cc:487] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:49.857614 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:49.857748 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:49.858002 446 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:49.858451 1178 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:49.858593 1178 ts_tablet_manager.cc:1397] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.000s
I20250112 20:51:49.859263 1178 raft_consensus.cc:357] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.859447 1178 raft_consensus.cc:383] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.859529 1178 raft_consensus.cc:738] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.859735 1178 consensus_queue.cc:260] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.860330 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8923b3ba7b44d080c19cd0e2103c49 (DEFAULT_TABLE table=default.Table_7e689916a41242d4bc7eec11c33be6b5 [id=c648a25ab5b14a25a7b1ed5c10ba4f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.860643 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8923b3ba7b44d080c19cd0e2103c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.863528 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.863683 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ced81345e31a4634bb21e3702ad69c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:49.863997 891 tablet_service.cc:1467] Processing CreateTablet for tablet 544ea9d983794cf69ec68d96ec5e0438 (DEFAULT_TABLE table=default.Table_6188282bedce4235988a0602b6addcaa [id=6c1e08f27aff4cfb8b3b09eeda4c3d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.864413 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544ea9d983794cf69ec68d96ec5e0438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.865060 880 tablet_service.cc:1467] Processing CreateTablet for tablet 752cd0d20fef49ad9bcfc0d3cbec8430 (DEFAULT_TABLE table=default.Table_737195573b924b53b18c84b5b0f7f44c [id=3252a9a2ddb440d5a7ee6236d4a95e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.865209 3142 raft_consensus.cc:2798] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.865314 3142 raft_consensus.cc:491] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 20:51:49.865275 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 53510.4us
I20250112 20:51:49.865376 3142 raft_consensus.cc:3054] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.865386 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 752cd0d20fef49ad9bcfc0d3cbec8430. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:49.865514 710 consensus_peers.cc:487] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 924082716c1e4d9ea6fbe97ae93232bd. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:49.867949 3142 raft_consensus.cc:513] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.868505 3142 leader_election.cc:290] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.865209 1164 raft_consensus.cc:357] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.868813 1164 raft_consensus.cc:383] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.868855 1164 raft_consensus.cc:738] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.869045 1164 consensus_queue.cc:260] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.879765 3121 raft_consensus.cc:2798] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:49.879889 3121 raft_consensus.cc:491] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:49.879963 3121 raft_consensus.cc:3054] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.880355 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7623bebb539e485aa508e95a35225979 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eb4ed0e18e7444c9173626df8b305b8) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.880617 710 leader_election.cc:304] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:49.882884 3121 raft_consensus.cc:513] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.883399 2447 ts_tablet_manager.cc:1918] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:49.883414 3121 leader_election.cc:290] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.883505 2447 log.cc:1198] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d0018e1eabf24241a358a7e00b6b5907
I20250112 20:51:49.883777 2447 ts_tablet_manager.cc:1939] T d0018e1eabf24241a358a7e00b6b5907 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.884040 3144 raft_consensus.cc:2798] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.884163 3144 raft_consensus.cc:695] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:49.884326 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c3812f8aa9ba422f85f331648a1ec02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.884441 3144 consensus_queue.cc:237] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.884796 1164 ts_tablet_manager.cc:1428] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20250112 20:51:49.887574 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c90c8dd4dfb14507accb82828be512f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.893024 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 53551931b23d43da89e9193cdf034a63 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:49.894878 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_557ef80f7a9c4980aa3aa81e9000ddf7" } modify_external_catalogs: true
I20250112 20:51:49.895042 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_557ef80f7a9c4980aa3aa81e9000ddf7" } modify_external_catalogs: true
I20250112 20:51:49.896190 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 3ea9425f3e7b46288c81dc8cdade99b2 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:49.896489 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 935) took 2348 ms. Trace:
I20250112 20:51:49.896569 2447 rpcz_store.cc:276] 0112 20:51:47.547559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.547595 (+ 36us) service_pool.cc:225] Handling call
0112 20:51:49.783528 (+2235933us) ts_tablet_manager.cc:1145] Deleting tablet d0018e1eabf24241a358a7e00b6b5907
0112 20:51:49.783537 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.834596 (+ 51059us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.883309 (+ 48713us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.896467 (+ 13158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235799}
I20250112 20:51:49.896855 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8523098ce690434c99af052e3170ca62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:49.897020 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 945) took 2126 ms. Trace:
I20250112 20:51:49.897083 2447 rpcz_store.cc:276] 0112 20:51:47.770956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.771041 (+ 85us) service_pool.cc:225] Handling call
0112 20:51:49.896910 (+2125869us) ts_tablet_manager.cc:1145] Deleting tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8
0112 20:51:49.896919 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.897007 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125658}
W20250112 20:51:49.897379 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8523098ce690434c99af052e3170ca62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.899511 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_04fff3dcd9d6433e85abc01fd00c77c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.899976 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04fff3dcd9d6433e85abc01fd00c77c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.900807 2447 tablet_replica.cc:331] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:49.901028 2447 raft_consensus.cc:2238] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.901194 2447 raft_consensus.cc:2267] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.901813 737 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd4177f9394a5a82d4f5e02369b3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.903386 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 577dc789f0cb48f49168124b37a6fba0 (table default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]) successfully deleted
I20250112 20:51:49.911177 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.911830 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.913226 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a5819f1c6329433183492855c3b08476
I20250112 20:51:49.913401 739 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.913691 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5b10576bd34af5b02f8d8dcc3759ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.914060 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bea4d83ea644a0930a29f61e7a5fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.914942 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.915675 753 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.916033 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.916358 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f875f075e98e45499291b519caf03184 with delete_type TABLET_DATA_DELETED (replaced by tablet 2363610c772b4a92bce5d4428f5f5c8e) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.916934 1163 tablet_bootstrap.cc:492] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:49.917080 1163 ts_tablet_manager.cc:1397] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.001s
I20250112 20:51:49.917737 3121 consensus_queue.cc:1035] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:49.919834 1163 raft_consensus.cc:357] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.920022 1163 raft_consensus.cc:383] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.920075 1163 raft_consensus.cc:738] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.920212 2447 ts_tablet_manager.cc:1905] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.890799 3194 raft_consensus.cc:491] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.922600 3194 raft_consensus.cc:513] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.923168 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.920287 1163 consensus_queue.cc:260] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.923483 3194 leader_election.cc:290] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:49.923828 1163 ts_tablet_manager.cc:1428] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:49.925170 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 53551931b23d43da89e9193cdf034a63
I20250112 20:51:49.925585 453 catalog_manager.cc:6596] ReplaceTablet: tablet 53551931b23d43da89e9193cdf034a63 deleted and replaced by tablet f1b0c67644464da4b4ce3aaf7d1071c7
I20250112 20:51:49.926446 3085 tablet.cc:1721] T dea98c29bc744009ab9b3eeb53684f29 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.897759 1178 ts_tablet_manager.cc:1428] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250112 20:51:49.928275 3078 tablet.cc:1721] T 161951bfabff485dbf7dceeac2d4128d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:49.900682 847 leader_election.cc:304] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:49.929436 3194 raft_consensus.cc:2798] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:49.929783 3194 raft_consensus.cc:695] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:49.930073 3194 consensus_queue.cc:237] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.932235 1163 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:49.933140 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:49.933398 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:49.933394 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9880e7d303b54d959bca4e8464a6b30c" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:49.933530 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:49.933589 758 raft_consensus.cc:3054] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.933753 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:49.933905 1163 tablet_bootstrap.cc:654] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.934374 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_cb2b5abd46bc4877887604a94159e418 [id=d46a20ad8b99492395771d94f5dc60fb] alter complete (version 2)
I20250112 20:51:49.934922 1178 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:49.936218 758 raft_consensus.cc:2463] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:49.941222 750 tablet_service.cc:1514] Processing DeleteTablet for tablet ced81345e31a4634bb21e3702ad69c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.944911 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a5819f1c6329433183492855c3b08476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.945473 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet c3812f8aa9ba422f85f331648a1ec02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.946372 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 53551931b23d43da89e9193cdf034a63 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b0c67644464da4b4ce3aaf7d1071c7) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:49.949419 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 2e6b4882a3d744bda5677aa9e01a53f4 (table default.Table_77909cafdcc948d7804b6f8b05ad48e8 [id=528071eaaf05426ea8cfed3779185b80]) successfully deleted
I20250112 20:51:49.949733 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d0018e1eabf24241a358a7e00b6b5907 (table default.Table_6d7de66040254347bff24c5702f6241c [id=45c1a2f4e5fa4a6f95a955f54e03f4a4]) successfully deleted
W20250112 20:51:49.949929 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 already in progress: creating tablet
I20250112 20:51:49.953002 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet a5819f1c6329433183492855c3b08476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.957221 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:33 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.913298 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4bd27cc30c422d85470c5a2df5d3ab (DEFAULT_TABLE table=default.Table_6763d75a41e34bc7aee13341771b4c7e [id=e94da089ebf9424bbd5017d1a4da65c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:49.958482 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 53551931b23d43da89e9193cdf034a63 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b0c67644464da4b4ce3aaf7d1071c7) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.959280 3097 tablet.cc:1721] T dea98c29bc744009ab9b3eeb53684f29 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:49.962571 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 9880e7d303b54d959bca4e8464a6b30c with delete_type TABLET_DATA_DELETED (replaced by tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.965147 1164 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.966841 1164 tablet_bootstrap.cc:654] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:49.967440 3121 raft_consensus.cc:491] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.967635 3121 raft_consensus.cc:513] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.968526 3121 leader_election.cc:290] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.972652 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4bd27cc30c422d85470c5a2df5d3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:49.975252 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 974) took 2183 ms. Trace:
I20250112 20:51:49.975373 1185 rpcz_store.cc:276] 0112 20:51:47.791911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.791952 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:49.530742 (+1738790us) ts_tablet_manager.cc:1145] Deleting tablet 577dc789f0cb48f49168124b37a6fba0
0112 20:51:49.530752 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.634820 (+104068us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.817877 (+183057us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.975204 (+157327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738675}
I20250112 20:51:49.975823 1185 tablet_replica.cc:331] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:49.976220 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 577dc789f0cb48f49168124b37a6fba0 (table default.Table_141fc45760534923b67d082dae64762d [id=29807af756ff4966a102c2b3aefc060d]) successfully deleted
I20250112 20:51:49.976503 3121 raft_consensus.cc:491] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:49.976715 3121 raft_consensus.cc:513] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:49.977653 3121 leader_election.cc:290] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:49.978578 2447 ts_tablet_manager.cc:1918] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.980130 3078 tablet.cc:1721] T cc90d3722fd14d59b9994655cd30ef2b P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:49.980679 2447 log.cc:1198] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/226ea7f38f1445c598676715a3036227
I20250112 20:51:49.981349 2447 ts_tablet_manager.cc:1939] T 226ea7f38f1445c598676715a3036227 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:49.982877 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_43c23610617d4b979805a95633d6c29c" } new_table_name: "default.Table_418cbc0f13b942718efd3ac122db95ca" modify_external_catalogs: true
I20250112 20:51:49.984436 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_074b8d5f172f47cd856859c9b1e4c702"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:49.985064 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_074b8d5f172f47cd856859c9b1e4c702 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:49.986104 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 948) took 2214 ms. Trace:
I20250112 20:51:49.986493 1185 raft_consensus.cc:2238] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:49.986681 1185 raft_consensus.cc:2267] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:49.987231 1185 ts_tablet_manager.cc:1905] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:49.987968 2447 rpcz_store.cc:276] 0112 20:51:47.771236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.771285 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:49.897254 (+2125969us) ts_tablet_manager.cc:1145] Deleting tablet 226ea7f38f1445c598676715a3036227
0112 20:51:49.897262 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.954157 (+ 56895us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.978479 (+ 24322us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.986081 (+ 7602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125808}
I20250112 20:51:49.989245 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 226ea7f38f1445c598676715a3036227 (table default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]) successfully deleted
I20250112 20:51:49.990160 1164 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:49.990303 1164 ts_tablet_manager.cc:1397] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.006s
I20250112 20:51:49.991032 1164 raft_consensus.cc:357] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.991348 1164 raft_consensus.cc:383] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:49.991428 1164 raft_consensus.cc:738] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:49.991607 1164 consensus_queue.cc:260] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:49.992434 1164 ts_tablet_manager.cc:1428] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:49.883678 1186 ts_tablet_manager.cc:1918] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:49.992592 1186 log.cc:1198] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/98547168378947df885c601ccdc85e57
I20250112 20:51:49.992688 1164 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:49.993037 1186 ts_tablet_manager.cc:1939] T 98547168378947df885c601ccdc85e57 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:49.920574 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ced81345e31a4634bb21e3702ad69c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.920642 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c90c8dd4dfb14507accb82828be512f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:49.994446 1164 tablet_bootstrap.cc:654] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:49.917762 710 leader_election.cc:343] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:49.995867 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 78079.6us
W20250112 20:51:49.996188 710 leader_election.cc:343] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:49.997478 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549d92191a594788bf639cd60a6b5142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.997733 906 raft_consensus.cc:3054] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.998116 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:49.998342 908 raft_consensus.cc:3054] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:49.999382 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80791d8582446b89adf212d3d6507eb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.000932 876 tablet_service.cc:1514] Processing DeleteTablet for tablet c3812f8aa9ba422f85f331648a1ec02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.001174 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7623bebb539e485aa508e95a35225979 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eb4ed0e18e7444c9173626df8b305b8) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.001753 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c90c8dd4dfb14507accb82828be512f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.001894 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 961a72a9d8a643cea743f3f021bd5a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.002264 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a5819f1c6329433183492855c3b08476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:49 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.002655 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 53551931b23d43da89e9193cdf034a63 with delete_type TABLET_DATA_DELETED (replaced by tablet f1b0c67644464da4b4ce3aaf7d1071c7) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:50.003247 847 leader_election.cc:343] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.004118 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.006377 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 946) took 2235 ms. Trace:
I20250112 20:51:50.006500 2447 rpcz_store.cc:276] 0112 20:51:47.771042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.771085 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:50.006254 (+2235169us) ts_tablet_manager.cc:1145] Deleting tablet 438e2c14353c4a8cb7242d75bb3ba673
0112 20:51:50.006264 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.006350 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2234609}
W20250112 20:51:50.006947 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 438e2c14353c4a8cb7242d75bb3ba673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438e2c14353c4a8cb7242d75bb3ba673 already in progress: creating tablet
I20250112 20:51:50.007367 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 949) took 2236 ms. Trace:
I20250112 20:51:50.007474 2447 rpcz_store.cc:276] 0112 20:51:47.771310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.771665 (+ 355us) service_pool.cc:225] Handling call
0112 20:51:50.007269 (+2235604us) ts_tablet_manager.cc:1145] Deleting tablet 6b440da020d947dda2553d65dd6a4bdf
0112 20:51:50.007278 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.007352 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235455}
W20250112 20:51:50.007894 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6b440da020d947dda2553d65dd6a4bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b440da020d947dda2553d65dd6a4bdf already in progress: creating tablet
I20250112 20:51:50.008236 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 942) took 2237 ms. Trace:
I20250112 20:51:50.008335 2447 rpcz_store.cc:276] 0112 20:51:47.770714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.776346 (+ 5632us) service_pool.cc:225] Handling call
0112 20:51:50.008147 (+2231801us) ts_tablet_manager.cc:1145] Deleting tablet 0bb9c2e6f37f4184a106aed05bae7924
0112 20:51:50.008155 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.008223 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231645}
W20250112 20:51:50.008702 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 0bb9c2e6f37f4184a106aed05bae7924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bb9c2e6f37f4184a106aed05bae7924 already in progress: creating tablet
I20250112 20:51:50.009068 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 943) took 2238 ms. Trace:
I20250112 20:51:50.009171 2447 rpcz_store.cc:276] 0112 20:51:47.770792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.776529 (+ 5737us) service_pool.cc:225] Handling call
0112 20:51:50.008981 (+2232452us) ts_tablet_manager.cc:1145] Deleting tablet 9705feb15f384b5185935eacc521d0e0
0112 20:51:50.008989 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.009055 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232352}
W20250112 20:51:50.009542 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 9705feb15f384b5185935eacc521d0e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9705feb15f384b5185935eacc521d0e0 already in progress: creating tablet
I20250112 20:51:50.009907 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 947) took 2238 ms. Trace:
I20250112 20:51:50.010016 2447 rpcz_store.cc:276] 0112 20:51:47.771151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.776706 (+ 5555us) service_pool.cc:225] Handling call
0112 20:51:50.009813 (+2233107us) ts_tablet_manager.cc:1145] Deleting tablet b062c1d67d4443eabe8ac1d8a832c243
0112 20:51:50.009821 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.009894 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232968}
W20250112 20:51:50.010730 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b062c1d67d4443eabe8ac1d8a832c243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b062c1d67d4443eabe8ac1d8a832c243 already in progress: creating tablet
I20250112 20:51:50.011121 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 944) took 2240 ms. Trace:
I20250112 20:51:50.011256 2447 rpcz_store.cc:276] 0112 20:51:47.770871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.776651 (+ 5780us) service_pool.cc:225] Handling call
0112 20:51:50.011028 (+2234377us) ts_tablet_manager.cc:1145] Deleting tablet 7d9572179a6f45759214c6c39f22b2f0
0112 20:51:50.011036 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.011105 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2234085}
I20250112 20:51:50.011839 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 955) took 2176 ms. Trace:
W20250112 20:51:50.011634 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7d9572179a6f45759214c6c39f22b2f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9572179a6f45759214c6c39f22b2f0 already in progress: creating tablet
I20250112 20:51:50.012255 2447 rpcz_store.cc:276] 0112 20:51:47.834939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.835173 (+ 234us) service_pool.cc:225] Handling call
0112 20:51:50.011754 (+2176581us) ts_tablet_manager.cc:1145] Deleting tablet 146994989d6d4ce8a12e1b8677d7feb7
0112 20:51:50.011762 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.011826 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066797}
W20250112 20:51:50.012639 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 146994989d6d4ce8a12e1b8677d7feb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146994989d6d4ce8a12e1b8677d7feb7 already in progress: creating tablet
I20250112 20:51:50.012965 2447 tablet_replica.cc:331] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.013185 2447 raft_consensus.cc:2238] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.013384 2447 raft_consensus.cc:2267] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.013932 2447 ts_tablet_manager.cc:1905] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.016113 1185 ts_tablet_manager.cc:1918] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:50.019464 906 raft_consensus.cc:2463] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.020052 2447 ts_tablet_manager.cc:1918] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:50.020291 2447 log.cc:1198] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/71743c73b6f24da9b14a3d4148117813
I20250112 20:51:50.020644 2447 ts_tablet_manager.cc:1939] T 71743c73b6f24da9b14a3d4148117813 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:50.021363 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 962) took 1839 ms. Trace:
I20250112 20:51:50.021483 2447 rpcz_store.cc:276] 0112 20:51:48.182227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.182273 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:50.012925 (+1830652us) ts_tablet_manager.cc:1145] Deleting tablet 71743c73b6f24da9b14a3d4148117813
0112 20:51:50.012934 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.016834 (+ 3900us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.019982 (+ 3148us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.021346 (+ 1364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830495}
I20250112 20:51:50.021910 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]) successfully deleted
I20250112 20:51:50.022287 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 963) took 1839 ms. Trace:
I20250112 20:51:50.022392 2447 rpcz_store.cc:276] 0112 20:51:48.182585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.182625 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:50.022187 (+1839562us) ts_tablet_manager.cc:1145] Deleting tablet 2c8c7784e5534c2c91f124dffd24d5c5
0112 20:51:50.022195 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.022274 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1838922}
W20250112 20:51:50.022784 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 2c8c7784e5534c2c91f124dffd24d5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8c7784e5534c2c91f124dffd24d5c5 already in progress: creating tablet
I20250112 20:51:50.023083 2447 tablet_replica.cc:331] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.023281 2447 raft_consensus.cc:2238] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.023296 1185 log.cc:1198] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/c8d24bccf682483aa9b273fc1ce847da
I20250112 20:51:50.023458 2447 raft_consensus.cc:2267] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.023617 1185 ts_tablet_manager.cc:1939] T c8d24bccf682483aa9b273fc1ce847da P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:50.023954 2447 ts_tablet_manager.cc:1905] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.025445 908 raft_consensus.cc:2463] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.026111 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 947) took 3508 ms. Trace:
I20250112 20:51:50.026208 1186 rpcz_store.cc:276] 0112 20:51:46.518004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.518412 (+ 408us) service_pool.cc:225] Handling call
0112 20:51:49.607059 (+3088647us) ts_tablet_manager.cc:1145] Deleting tablet 98547168378947df885c601ccdc85e57
0112 20:51:49.607070 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:49.754242 (+147172us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:49.883578 (+129336us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.026081 (+142503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088490}
I20250112 20:51:50.027860 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 956) took 3319 ms. Trace:
I20250112 20:51:50.028353 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_96d7128d31fe403abd7f96408718800c" } modify_external_catalogs: true
I20250112 20:51:50.028517 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_96d7128d31fe403abd7f96408718800c" } modify_external_catalogs: true
I20250112 20:51:50.028573 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 975) took 2236 ms. Trace:
I20250112 20:51:50.027189 1164 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:50.028599 1186 rpcz_store.cc:276] 0112 20:51:46.708679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.708725 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:50.027693 (+3318968us) ts_tablet_manager.cc:1145] Deleting tablet 6158b97ccf0a42fdb1a510c5d1e80213
0112 20:51:50.027713 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.027806 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3317539}
I20250112 20:51:50.029875 1186 tablet_replica.cc:331] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.030699 1186 raft_consensus.cc:2238] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.031091 2447 ts_tablet_manager.cc:1918] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:50.031266 2447 log.cc:1198] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:50.031635 2447 ts_tablet_manager.cc:1939] T 8c5b10576bd34af5b02f8d8dcc3759ab P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:50.028971 1164 ts_tablet_manager.cc:1397] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.001s
I20250112 20:51:50.032629 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 965) took 1631 ms. Trace:
I20250112 20:51:50.032806 2447 rpcz_store.cc:276] 0112 20:51:48.400868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.400902 (+ 34us) service_pool.cc:225] Handling call
0112 20:51:50.023047 (+1622145us) ts_tablet_manager.cc:1145] Deleting tablet 8c5b10576bd34af5b02f8d8dcc3759ab
0112 20:51:50.023055 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.026663 (+ 3608us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.031004 (+ 4341us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.032606 (+ 1602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614065}
I20250112 20:51:50.032938 1164 raft_consensus.cc:357] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.033226 1164 raft_consensus.cc:383] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.033316 1164 raft_consensus.cc:738] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.033576 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 8c5b10576bd34af5b02f8d8dcc3759ab (table default.Table_0bec6474ab5a44ad91c57be6219a3f10 [id=98968a56ccaa48f181010a22e64669ce]) successfully deleted
I20250112 20:51:50.033546 1164 consensus_queue.cc:260] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.034155 1164 ts_tablet_manager.cc:1428] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:50.034896 1164 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:50.034819 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 966) took 1633 ms. Trace:
I20250112 20:51:50.035516 2447 rpcz_store.cc:276] 0112 20:51:48.401014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.401052 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:50.034593 (+1633541us) ts_tablet_manager.cc:1145] Deleting tablet 921ae136d5e2438e950b96f156f23da3
0112 20:51:50.034605 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.034801 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625487}
W20250112 20:51:50.036036 440 catalog_manager.cc:4839] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 921ae136d5e2438e950b96f156f23da3 because the tablet was not found. No further retry: Not found: Tablet not found: 921ae136d5e2438e950b96f156f23da3
I20250112 20:51:50.036407 2447 tablet_replica.cc:331] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.036664 2447 raft_consensus.cc:2238] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.036921 2447 raft_consensus.cc:2267] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.037569 1164 tablet_bootstrap.cc:654] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.037715 2447 ts_tablet_manager.cc:1905] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.037803 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7623bebb539e485aa508e95a35225979 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eb4ed0e18e7444c9173626df8b305b8) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.028673 1185 rpcz_store.cc:276] 0112 20:51:47.791999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.792041 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:49.975739 (+2183698us) ts_tablet_manager.cc:1145] Deleting tablet c8d24bccf682483aa9b273fc1ce847da
0112 20:51:49.975752 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.013020 (+ 37268us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.016003 (+ 2983us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.028523 (+ 12520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183573}
I20250112 20:51:50.040335 1185 tablet_replica.cc:331] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:50.040382 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet c8d24bccf682483aa9b273fc1ce847da (table default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]) successfully deleted
I20250112 20:51:50.040848 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549d92191a594788bf639cd60a6b5142" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.041291 1185 raft_consensus.cc:2238] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.041500 1185 raft_consensus.cc:2267] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.047389 2447 ts_tablet_manager.cc:1918] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.047501 2447 log.cc:1198] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b4ef01b41b6140d6bdad07bc79dd7dc5
I20250112 20:51:50.047760 1178 tablet_bootstrap.cc:654] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.048559 1185 ts_tablet_manager.cc:1905] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.051021 3130 consensus_queue.cc:1035] T be4a237cfeeb4d0fb7ad1abb4ff97845 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.052188 2447 ts_tablet_manager.cc:1939] T b4ef01b41b6140d6bdad07bc79dd7dc5 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:50.052914 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 967) took 1651 ms. Trace:
I20250112 20:51:50.053000 2447 rpcz_store.cc:276] 0112 20:51:48.401105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.401133 (+ 28us) service_pool.cc:225] Handling call
0112 20:51:50.036353 (+1635220us) ts_tablet_manager.cc:1145] Deleting tablet b4ef01b41b6140d6bdad07bc79dd7dc5
0112 20:51:50.036366 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.043547 (+ 7181us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.047302 (+ 3755us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.052896 (+ 5594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627133}
I20250112 20:51:50.053378 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b4ef01b41b6140d6bdad07bc79dd7dc5 (table default.Table_0787fb4223774e1889d1d09fb80f87f5 [id=fb7c117897c24636b7e2dd8109e782d2]) successfully deleted
I20250112 20:51:50.053550 2447 tablet_replica.cc:331] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.053741 2447 raft_consensus.cc:2238] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.054977 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480dca3c3a1e4da7a84834e02f26b299" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.055331 2447 raft_consensus.cc:2267] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.055995 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480dca3c3a1e4da7a84834e02f26b299" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.056294 908 raft_consensus.cc:2463] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.057214 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.057629 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet dea98c29bc744009ab9b3eeb53684f29 (table default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264]). Expected version 2 got 0
I20250112 20:51:50.057829 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:50.058007 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.058102 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:50.058179 908 raft_consensus.cc:2463] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.058275 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:50.058573 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet e72a6c8e18934b789a048d5aaded121b (table default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa]). Expected version 2 got 0
I20250112 20:51:50.059327 1039 raft_consensus.cc:3054] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.062613 755 tablet_service.cc:1467] Processing CreateTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 (DEFAULT_TABLE table=default.Table_963ba961a67b40e08a07744599f1673d [id=fc6c4e199f2b4c64b33d49c4abef3daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.062860 741 tablet_service.cc:1467] Processing CreateTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb (DEFAULT_TABLE table=default.Table_11813bf0d59d467d8f57da28e92d1b9f [id=57d583a9d60343e2ad34be97370dece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.063063 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df292cbdf58b4fcd889fcb63335b37d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.063138 2923 raft_consensus.cc:491] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.063261 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.063304 2923 raft_consensus.cc:513] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.063900 2923 leader_election.cc:290] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.062637 756 tablet_service.cc:1467] Processing CreateTablet for tablet ccc7e9b557614de48a90814d4653d885 (DEFAULT_TABLE table=default.Table_95f1ea8dfc6a4580a7b027fcc460ca76 [id=891b6a83728c48008d052b51f866e468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.065261 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc7e9b557614de48a90814d4653d885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.066085 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.066279 908 raft_consensus.cc:2463] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.066967 2447 ts_tablet_manager.cc:1905] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.070267 710 leader_election.cc:304] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.071720 2923 raft_consensus.cc:491] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.071867 2923 raft_consensus.cc:513] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.072412 2923 leader_election.cc:290] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:50.072671 1040 raft_consensus.cc:3054] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.072968 2923 raft_consensus.cc:2798] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.073549 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.073709 1045 raft_consensus.cc:2463] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.073819 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb (DEFAULT_TABLE table=default.Table_11813bf0d59d467d8f57da28e92d1b9f [id=57d583a9d60343e2ad34be97370dece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.074136 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.073565 1039 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.076282 2923 raft_consensus.cc:695] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.076557 2923 consensus_queue.cc:237] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.076650 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9288838d762d4709862694f7eafb796b (DEFAULT_TABLE table=default.Table_7009bf4316cd40bb9c4d7ea687dd7d03 [id=9bd36d393f66423eb8b88eabf40c9fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.077952 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9288838d762d4709862694f7eafb796b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.077899 1013 tablet_service.cc:1467] Processing CreateTablet for tablet fcf88720596d46628ef1fbd6737582d6 (DEFAULT_TABLE table=default.Table_36ebfe68c6e34958b6283665c113c466 [id=683e20944f214153a9693d397f3d4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.078222 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcf88720596d46628ef1fbd6737582d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.078787 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ccc7e9b557614de48a90814d4653d885 (DEFAULT_TABLE table=default.Table_95f1ea8dfc6a4580a7b027fcc460ca76 [id=891b6a83728c48008d052b51f866e468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.079079 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc7e9b557614de48a90814d4653d885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.081010 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9288838d762d4709862694f7eafb796b (DEFAULT_TABLE table=default.Table_7009bf4316cd40bb9c4d7ea687dd7d03 [id=9bd36d393f66423eb8b88eabf40c9fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.081444 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9288838d762d4709862694f7eafb796b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.081944 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3f7793320d1b43deb2bc7e107686169f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.082458 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7793320d1b43deb2bc7e107686169f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.086918 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2050ea3fd15848d7b59a8b8020368824"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:50.087114 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8aba8733d71c40e690d75bc533ebddb3" } modify_external_catalogs: true
I20250112 20:51:50.087271 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8aba8733d71c40e690d75bc533ebddb3" } modify_external_catalogs: true
W20250112 20:51:50.087572 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2050ea3fd15848d7b59a8b8020368824 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.088217 1018 tablet_service.cc:1467] Processing CreateTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 (DEFAULT_TABLE table=default.Table_963ba961a67b40e08a07744599f1673d [id=fc6c4e199f2b4c64b33d49c4abef3daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.088667 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df292cbdf58b4fcd889fcb63335b37d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.093502 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2ac8449ec8b94e0494cceb60303f0481" } new_table_name: "default.Table_462228fad099412985e60ba1141be1dd" modify_external_catalogs: true
I20250112 20:51:50.094985 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c36b678fdc7544dc9acc25643ccab97c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.095578 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c36b678fdc7544dc9acc25643ccab97c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.096252 1185 ts_tablet_manager.cc:1918] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.096383 1185 log.cc:1198] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1fd34e829d0645ea80daaff5d27cb266
I20250112 20:51:50.096853 1185 ts_tablet_manager.cc:1939] T 1fd34e829d0645ea80daaff5d27cb266 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:50.097615 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 977) took 2305 ms. Trace:
I20250112 20:51:50.097708 1185 rpcz_store.cc:276] 0112 20:51:47.792162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.792192 (+ 30us) service_pool.cc:225] Handling call
0112 20:51:50.040244 (+2248052us) ts_tablet_manager.cc:1145] Deleting tablet 1fd34e829d0645ea80daaff5d27cb266
0112 20:51:50.040257 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.055588 (+ 15331us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.096140 (+ 40552us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.097573 (+ 1433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2247950}
I20250112 20:51:50.098191 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 991) took 2233 ms. Trace:
I20250112 20:51:50.098282 1185 rpcz_store.cc:276] 0112 20:51:47.864652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.869383 (+ 4731us) service_pool.cc:225] Handling call
0112 20:51:50.098020 (+2228637us) ts_tablet_manager.cc:1145] Deleting tablet 0bb9c2e6f37f4184a106aed05bae7924
0112 20:51:50.098030 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.098151 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228439}
I20250112 20:51:50.098626 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 992) took 2233 ms. Trace:
I20250112 20:51:50.098717 1185 rpcz_store.cc:276] 0112 20:51:47.864722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.869566 (+ 4844us) service_pool.cc:225] Handling call
0112 20:51:50.098489 (+2228923us) ts_tablet_manager.cc:1145] Deleting tablet 9705feb15f384b5185935eacc521d0e0
0112 20:51:50.098499 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.098576 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228801}
I20250112 20:51:50.099056 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 994) took 2225 ms. Trace:
I20250112 20:51:50.099139 1185 rpcz_store.cc:276] 0112 20:51:47.873235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.873277 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:50.098938 (+2225661us) ts_tablet_manager.cc:1145] Deleting tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8
0112 20:51:50.098946 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.099017 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225524}
I20250112 20:51:50.099429 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 995) took 2226 ms. Trace:
I20250112 20:51:50.099526 1185 rpcz_store.cc:276] 0112 20:51:47.873320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.873493 (+ 173us) service_pool.cc:225] Handling call
0112 20:51:50.099315 (+2225822us) ts_tablet_manager.cc:1145] Deleting tablet 438e2c14353c4a8cb7242d75bb3ba673
0112 20:51:50.099323 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.099393 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225692}
I20250112 20:51:50.099849 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 993) took 2235 ms. Trace:
I20250112 20:51:50.099948 1185 rpcz_store.cc:276] 0112 20:51:47.864804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.873107 (+ 8303us) service_pool.cc:225] Handling call
0112 20:51:50.099719 (+2226612us) ts_tablet_manager.cc:1145] Deleting tablet 7d9572179a6f45759214c6c39f22b2f0
0112 20:51:50.099727 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.099812 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2198266}
I20250112 20:51:50.100283 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 996) took 2186 ms. Trace:
I20250112 20:51:50.100363 1185 rpcz_store.cc:276] 0112 20:51:47.913535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:47.913584 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:50.100169 (+2186585us) ts_tablet_manager.cc:1145] Deleting tablet b062c1d67d4443eabe8ac1d8a832c243
0112 20:51:50.100176 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.100244 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183150}
I20250112 20:51:50.100698 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1000) took 2093 ms. Trace:
I20250112 20:51:50.103446 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ec6d004e89864f5bb239592c66002af8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.103919 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec6d004e89864f5bb239592c66002af8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.098738 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1fd34e829d0645ea80daaff5d27cb266 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
W20250112 20:51:50.104398 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 0bb9c2e6f37f4184a106aed05bae7924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bb9c2e6f37f4184a106aed05bae7924 already in progress: creating tablet
W20250112 20:51:50.104602 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 9705feb15f384b5185935eacc521d0e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9705feb15f384b5185935eacc521d0e0 already in progress: creating tablet
W20250112 20:51:50.104758 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 already in progress: creating tablet
W20250112 20:51:50.106189 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 438e2c14353c4a8cb7242d75bb3ba673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438e2c14353c4a8cb7242d75bb3ba673 already in progress: creating tablet
W20250112 20:51:50.106348 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 7d9572179a6f45759214c6c39f22b2f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9572179a6f45759214c6c39f22b2f0 already in progress: creating tablet
W20250112 20:51:50.106508 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b062c1d67d4443eabe8ac1d8a832c243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b062c1d67d4443eabe8ac1d8a832c243 already in progress: creating tablet
I20250112 20:51:50.109052 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c1535f7854fa43009733487f7b3af237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.109524 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1535f7854fa43009733487f7b3af237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.110615 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b043070379134d4bb916ee883c48fccd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.111030 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b043070379134d4bb916ee883c48fccd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.112030 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_22606a15a61946b08eec7776233747ed" } new_table_name: "default.Table_89eea9d803154ffb8eb6e6b2e8558558" modify_external_catalogs: true
W20250112 20:51:50.113919 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 752cd0d20fef49ad9bcfc0d3cbec8430 (table default.Table_7f87f7dbd08e4a5abb17ed757bbe20de [id=3252a9a2ddb440d5a7ee6236d4a95e42], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 752cd0d20fef49ad9bcfc0d3cbec8430 consensus state has no leader
I20250112 20:51:50.119143 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8bc545822e98460e815df4e6cad455b4
W20250112 20:51:50.121193 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 117744us
I20250112 20:51:50.122309 1186 raft_consensus.cc:2267] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.122924 1186 ts_tablet_manager.cc:1905] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.123811 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e55e24b020a64e9ebaf27b249838c000" } modify_external_catalogs: true
I20250112 20:51:50.123965 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e55e24b020a64e9ebaf27b249838c000" } modify_external_catalogs: true
I20250112 20:51:50.126827 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_638524600a3c447b90e0c84bf8d929c5" } modify_external_catalogs: true
I20250112 20:51:50.126967 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_638524600a3c447b90e0c84bf8d929c5" } modify_external_catalogs: true
I20250112 20:51:50.129760 1186 ts_tablet_manager.cc:1918] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.130210 1186 log.cc:1198] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8229e4ef771b4032a4bfae3190a851e3
I20250112 20:51:50.130739 1186 ts_tablet_manager.cc:1939] T 8229e4ef771b4032a4bfae3190a851e3 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.131177 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 957) took 3420 ms. Trace:
I20250112 20:51:50.131367 1186 rpcz_store.cc:276] 0112 20:51:46.711127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.711178 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:50.029814 (+3318636us) ts_tablet_manager.cc:1145] Deleting tablet 8229e4ef771b4032a4bfae3190a851e3
0112 20:51:50.029823 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.125978 (+ 96155us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.129639 (+ 3661us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.131141 (+ 1502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90876,"tablet-delete.queue_time_us":3318465}
I20250112 20:51:50.131996 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 958) took 3420 ms. Trace:
I20250112 20:51:50.132170 1186 rpcz_store.cc:276] 0112 20:51:46.711487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.711528 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:50.131827 (+3420299us) ts_tablet_manager.cc:1145] Deleting tablet 2de03e3114d145aeb81590dbae2c548e
0112 20:51:50.131839 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.131960 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3420150}
I20250112 20:51:50.132565 1186 tablet_replica.cc:331] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.132649 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 98547168378947df885c601ccdc85e57 (table default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]) successfully deleted
W20250112 20:51:50.132894 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6158b97ccf0a42fdb1a510c5d1e80213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6158b97ccf0a42fdb1a510c5d1e80213 already in progress: creating tablet
I20250112 20:51:50.132956 1186 raft_consensus.cc:2238] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.134025 1186 raft_consensus.cc:2267] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.134722 1186 ts_tablet_manager.cc:1905] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.143172 1186 ts_tablet_manager.cc:1918] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:50.143436 1186 log.cc:1198] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/61b74aaf7f5b4c87acbefaa4df5c10d2
I20250112 20:51:50.143868 1186 ts_tablet_manager.cc:1939] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.144256 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 959) took 3387 ms. Trace:
I20250112 20:51:50.144408 1186 rpcz_store.cc:276] 0112 20:51:46.756808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:46.756877 (+ 69us) service_pool.cc:225] Handling call
0112 20:51:50.132494 (+3375617us) ts_tablet_manager.cc:1145] Deleting tablet 61b74aaf7f5b4c87acbefaa4df5c10d2
0112 20:51:50.132505 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.139327 (+ 6822us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.143060 (+ 3733us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.144216 (+ 1156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":3375352}
I20250112 20:51:50.150228 1163 tablet_bootstrap.cc:492] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:50.150377 1163 ts_tablet_manager.cc:1397] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.218s user 0.006s sys 0.004s
I20250112 20:51:50.151779 1163 raft_consensus.cc:357] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.152025 1163 raft_consensus.cc:383] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.152089 1163 raft_consensus.cc:738] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.152318 1163 consensus_queue.cc:260] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.152848 1163 ts_tablet_manager.cc:1428] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 20:51:50.153100 1163 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:50.154472 1163 tablet_bootstrap.cc:654] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.163422 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5065a94bb13c419e9d9b8eaca72f5052
I20250112 20:51:50.166414 885 tablet_service.cc:1467] Processing CreateTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 (DEFAULT_TABLE table=default.Table_963ba961a67b40e08a07744599f1673d [id=fc6c4e199f2b4c64b33d49c4abef3daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.166798 3097 tablet.cc:1721] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:50.167026 888 tablet_service.cc:1467] Processing CreateTablet for tablet ccc7e9b557614de48a90814d4653d885 (DEFAULT_TABLE table=default.Table_95f1ea8dfc6a4580a7b027fcc460ca76 [id=891b6a83728c48008d052b51f866e468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.167563 882 tablet_service.cc:1467] Processing CreateTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb (DEFAULT_TABLE table=default.Table_11813bf0d59d467d8f57da28e92d1b9f [id=57d583a9d60343e2ad34be97370dece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.168170 884 tablet_service.cc:1467] Processing CreateTablet for tablet 9288838d762d4709862694f7eafb796b (DEFAULT_TABLE table=default.Table_7009bf4316cd40bb9c4d7ea687dd7d03 [id=9bd36d393f66423eb8b88eabf40c9fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.168514 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc545822e98460e815df4e6cad455b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.169224 879 tablet_service.cc:1467] Processing CreateTablet for tablet fcf88720596d46628ef1fbd6737582d6 (DEFAULT_TABLE table=default.Table_36ebfe68c6e34958b6283665c113c466 [id=683e20944f214153a9693d397f3d4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.174131 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c137b3886d444880aee556714375e8d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.174701 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c137b3886d444880aee556714375e8d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.176714 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc545822e98460e815df4e6cad455b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.180936 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.181042 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 40dc4e83a19c408d9bc021089cb94c5c (DEFAULT_TABLE table=default.Table_127c6f1785a7428793ecea2c36cf7a2b [id=4763fcbcf7304f25ac6ce5f4440dbc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.181422 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40dc4e83a19c408d9bc021089cb94c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.185117 1013 tablet_service.cc:1467] Processing CreateTablet for tablet bd48e38905e848a2880adc495cb7da94 (DEFAULT_TABLE table=default.Table_594e1478361d4b56a02d64049420de56 [id=60b582ff2bd44074bd595ac0002b3724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.185580 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48e38905e848a2880adc495cb7da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.188928 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb4ed0e18e7444c9173626df8b305b8 (DEFAULT_TABLE table=default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.189296 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb4ed0e18e7444c9173626df8b305b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:50.194990 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c54516970d424f7cb28c72d8fe52b824 consensus state has no leader
I20250112 20:51:50.199994 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 56544c3b0cf140089baa678765a2eb79 (table default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81]). Expected version 2 got 0
I20250112 20:51:50.200563 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:50.203981 1178 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:50.204144 1178 ts_tablet_manager.cc:1397] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.269s user 0.004s sys 0.000s
I20250112 20:51:50.204905 1178 raft_consensus.cc:357] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.205087 1178 raft_consensus.cc:383] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.205247 1178 raft_consensus.cc:738] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.205462 1178 consensus_queue.cc:260] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.209270 1178 ts_tablet_manager.cc:1428] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:50.210880 1178 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:50.212532 1178 tablet_bootstrap.cc:654] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.213652 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 10646eb42a25426db083ca07fb95cd47 (DEFAULT_TABLE table=default.Table_f67bc41b96f8402bbefe2a5a5be8ffd5 [id=40e98d42ebfe4a4bbff9c7cf52fd9470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.214107 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10646eb42a25426db083ca07fb95cd47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:50.216156 983 consensus_peers.cc:487] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dba3d9315db4b9292288ee62c2244bc. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:50.220286 2447 ts_tablet_manager.cc:1918] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.220582 2447 log.cc:1198] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/49b8cc99742f4f9ba68eb1aa17cab248
I20250112 20:51:50.221459 2447 ts_tablet_manager.cc:1939] T 49b8cc99742f4f9ba68eb1aa17cab248 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:50.222357 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 969) took 1821 ms. Trace:
I20250112 20:51:50.222510 2447 rpcz_store.cc:276] 0112 20:51:48.401305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.401350 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:50.053242 (+1651892us) ts_tablet_manager.cc:1145] Deleting tablet 49b8cc99742f4f9ba68eb1aa17cab248
0112 20:51:50.053252 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.130955 (+ 77703us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.220168 (+ 89213us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.222333 (+ 2165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643939}
I20250112 20:51:50.223975 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8229e4ef771b4032a4bfae3190a851e3 (table default.Table_11e0da2a1beb4196933adeeb88dfed4a [id=52b7e77f657e45af8b216c0bc7301ced]) successfully deleted
W20250112 20:51:50.224264 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2de03e3114d145aeb81590dbae2c548e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de03e3114d145aeb81590dbae2c548e already in progress: creating tablet
I20250112 20:51:50.226686 875 tablet_service.cc:1467] Processing CreateTablet for tablet bd48e38905e848a2880adc495cb7da94 (DEFAULT_TABLE table=default.Table_594e1478361d4b56a02d64049420de56 [id=60b582ff2bd44074bd595ac0002b3724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.227006 3178 consensus_queue.cc:1035] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.227381 891 tablet_service.cc:1467] Processing CreateTablet for tablet 10646eb42a25426db083ca07fb95cd47 (DEFAULT_TABLE table=default.Table_f67bc41b96f8402bbefe2a5a5be8ffd5 [id=40e98d42ebfe4a4bbff9c7cf52fd9470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.227954 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb4ed0e18e7444c9173626df8b305b8 (DEFAULT_TABLE table=default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.227991 3097 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.228456 889 tablet_service.cc:1467] Processing CreateTablet for tablet 40dc4e83a19c408d9bc021089cb94c5c (DEFAULT_TABLE table=default.Table_127c6f1785a7428793ecea2c36cf7a2b [id=4763fcbcf7304f25ac6ce5f4440dbc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.228814 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:50.233991 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02204076f2f44e369a74fe14680538b3 consensus state has no leader
I20250112 20:51:50.235078 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_799f579511f14383a50065deef83f536" } new_table_name: "default.Table_10eb2e0fd11e4f18802c103171f73926" modify_external_catalogs: true
I20250112 20:51:50.236544 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_df950a6830fa45b1a8f21feda53ea0ea" } new_table_name: "default.Table_2c9c479263b44516bafb19e7cd06c176" modify_external_catalogs: true
I20250112 20:51:50.237833 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_41daf866320a48988a8961ef716b8cf0" } modify_external_catalogs: true
I20250112 20:51:50.237988 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_41daf866320a48988a8961ef716b8cf0" } modify_external_catalogs: true
I20250112 20:51:50.240473 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e2436c1c8e5b4795a868b7f537315d19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.241039 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2436c1c8e5b4795a868b7f537315d19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.265343 1178 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:50.265581 1178 ts_tablet_manager.cc:1397] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.003s
I20250112 20:51:50.266629 1178 raft_consensus.cc:357] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.266950 1178 raft_consensus.cc:383] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.267038 1178 raft_consensus.cc:738] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.267354 1178 consensus_queue.cc:260] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.269702 1178 ts_tablet_manager.cc:1428] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:50.280588 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6181b166b5cb429da3ebba23cf267942
I20250112 20:51:50.281324 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c544fe5495bb4037970dfebccbbe09b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.281824 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c544fe5495bb4037970dfebccbbe09b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.282684 1164 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:50.283241 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet db9178cf782d4bd28eba90cc03796885
I20250112 20:51:50.283888 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b0e136fb45d404eb7351439ce056024" } new_table_name: "default.Table_5d120fb8bfe643a086111ad176916425" modify_external_catalogs: true
I20250112 20:51:50.290161 2886 raft_consensus.cc:491] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.290817 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.291060 1164 ts_tablet_manager.cc:1397] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.248s user 0.004s sys 0.000s
W20250112 20:51:50.291417 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 118627us
I20250112 20:51:50.291962 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.293473 2886 raft_consensus.cc:513] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.294869 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc545822e98460e815df4e6cad455b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.295477 710 leader_election.cc:304] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.296005 2886 leader_election.cc:290] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.297271 3121 raft_consensus.cc:2798] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.297593 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_e4a5db71ae7a4a31a4648186d20c3324 [id=cfaa5f6221c1491b822a803a83e8f264] alter complete (version 2)
I20250112 20:51:50.298084 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1a006bd6186c4ed89c0a6510ab5ff525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.298638 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a006bd6186c4ed89c0a6510ab5ff525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.301141 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb4ed0e18e7444c9173626df8b305b8 (DEFAULT_TABLE table=default.Table_f5d19e04b96b479d9c07282d7bb58644 [id=5d72d7982b2548f698e861fd6117a148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.301643 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb4ed0e18e7444c9173626df8b305b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.302021 750 tablet_service.cc:1467] Processing CreateTablet for tablet 10646eb42a25426db083ca07fb95cd47 (DEFAULT_TABLE table=default.Table_f67bc41b96f8402bbefe2a5a5be8ffd5 [id=40e98d42ebfe4a4bbff9c7cf52fd9470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.302410 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10646eb42a25426db083ca07fb95cd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.307231 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6181b166b5cb429da3ebba23cf267942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.307459 878 tablet_service.cc:1514] Processing DeleteTablet for tablet db9178cf782d4bd28eba90cc03796885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.308810 3121 raft_consensus.cc:695] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.309136 3121 consensus_queue.cc:237] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.303352 710 leader_election.cc:304] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.311498 756 tablet_service.cc:1467] Processing CreateTablet for tablet bd48e38905e848a2880adc495cb7da94 (DEFAULT_TABLE table=default.Table_594e1478361d4b56a02d64049420de56 [id=60b582ff2bd44074bd595ac0002b3724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.311916 755 tablet_service.cc:1467] Processing CreateTablet for tablet fcf88720596d46628ef1fbd6737582d6 (DEFAULT_TABLE table=default.Table_36ebfe68c6e34958b6283665c113c466 [id=683e20944f214153a9693d397f3d4e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.315030 742 tablet_service.cc:1467] Processing CreateTablet for tablet 40dc4e83a19c408d9bc021089cb94c5c (DEFAULT_TABLE table=default.Table_127c6f1785a7428793ecea2c36cf7a2b [id=4763fcbcf7304f25ac6ce5f4440dbc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.315395 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 5065a94bb13c419e9d9b8eaca72f5052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.315564 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6181b166b5cb429da3ebba23cf267942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.315732 744 tablet_service.cc:1514] Processing DeleteTablet for tablet db9178cf782d4bd28eba90cc03796885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:50.315856 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2be779fac1e4240a943e7862419bb82 consensus state has no leader
I20250112 20:51:50.318708 2886 raft_consensus.cc:2798] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.319135 1185 rpcz_store.cc:276] 0112 20:51:48.007353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.007441 (+ 88us) service_pool.cc:225] Handling call
0112 20:51:50.100566 (+2093125us) ts_tablet_manager.cc:1145] Deleting tablet 6b440da020d947dda2553d65dd6a4bdf
0112 20:51:50.100574 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.100658 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2091535}
I20250112 20:51:50.225921 3085 tablet.cc:1721] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:50.297648 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 6181b166b5cb429da3ebba23cf267942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.301883 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.302990 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 104d6b4effb84314a6b2b9db883b4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.303205 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f264d84ae4846d1afaaabc41d5789f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f424288fe62849edbbd2d38e89331fb7) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.322468 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 49b8cc99742f4f9ba68eb1aa17cab248 (table default.Table_25113f809e324283a5b53261123398f1 [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]) successfully deleted
W20250112 20:51:50.322870 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6b440da020d947dda2553d65dd6a4bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b440da020d947dda2553d65dd6a4bdf already in progress: creating tablet
I20250112 20:51:50.323591 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48e38905e848a2880adc495cb7da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.324609 710 leader_election.cc:304] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.324872 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f7b722b0eb5944a4b344a3833905a2b0
I20250112 20:51:50.325145 3144 raft_consensus.cc:2798] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.257491 3250 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.330430 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.332566 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fae6a2fd2e041f6953157f01d134e04" } modify_external_catalogs: true
I20250112 20:51:50.332741 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8fae6a2fd2e041f6953157f01d134e04" } modify_external_catalogs: true
I20250112 20:51:50.303561 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.334622 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_eefdbb79b5994581b1e544fa46ce498b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:50.335294 451 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
W20250112 20:51:50.335364 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eefdbb79b5994581b1e544fa46ce498b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.303702 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet db9178cf782d4bd28eba90cc03796885 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.312983 1163 tablet_bootstrap.cc:492] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:50.338387 1163 ts_tablet_manager.cc:1397] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.185s user 0.000s sys 0.004s
I20250112 20:51:50.338471 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 964) took 1937 ms. Trace:
I20250112 20:51:50.338704 2447 rpcz_store.cc:276] 0112 20:51:48.400739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.400824 (+ 85us) service_pool.cc:225] Handling call
0112 20:51:50.320219 (+1919395us) ts_tablet_manager.cc:1145] Deleting tablet cfb833659c23431daaab008dcdf1de43
0112 20:51:50.320229 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.338428 (+ 18199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910806}
I20250112 20:51:50.339468 2447 tablet_replica.cc:331] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.340029 2447 raft_consensus.cc:2238] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:50.340934 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet cfb833659c23431daaab008dcdf1de43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb833659c23431daaab008dcdf1de43 already in progress: creating tablet
I20250112 20:51:50.345327 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df292cbdf58b4fcd889fcb63335b37d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.349730 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd48e38905e848a2880adc495cb7da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.350498 2886 raft_consensus.cc:491] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.352417 1185 tablet_replica.cc:331] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:50.352819 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40dc4e83a19c408d9bc021089cb94c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.352872 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10646eb42a25426db083ca07fb95cd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.353102 3144 raft_consensus.cc:491] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.353276 1164 raft_consensus.cc:357] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.354089 710 leader_election.cc:304] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.354547 2632 raft_consensus.cc:2798] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.355357 2886 raft_consensus.cc:3054] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.355414 3144 raft_consensus.cc:3054] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.356495 2923 consensus_queue.cc:1035] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 20:51:50.356930 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.357254 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb4ed0e18e7444c9173626df8b305b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.357353 1185 raft_consensus.cc:2238] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.357561 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcf88720596d46628ef1fbd6737582d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.357740 1185 raft_consensus.cc:2267] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.358677 1185 ts_tablet_manager.cc:1905] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.359936 2757 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39271
I20250112 20:51:50.360883 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc7e9b557614de48a90814d4653d885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.364806 2632 raft_consensus.cc:491] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.364946 2632 raft_consensus.cc:3054] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.367381 2632 raft_consensus.cc:513] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.367640 2757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/master-1/data/info.pb
I20250112 20:51:50.367406 3256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39271 every 8 connection(s)
I20250112 20:51:50.368070 2632 leader_election.cc:290] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.368178 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9288838d762d4709862694f7eafb796b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.368832 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVmJ_bl/build/asan/bin/kudu as pid 2757
I20250112 20:51:50.340569 1163 raft_consensus.cc:357] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.373000 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.372995 1163 raft_consensus.cc:383] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.373111 1163 raft_consensus.cc:738] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.373360 1163 consensus_queue.cc:260] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.374034 1163 ts_tablet_manager.cc:1428] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20250112 20:51:50.374336 1163 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:50.375025 3144 raft_consensus.cc:513] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.375664 3144 leader_election.cc:290] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.375726 1163 tablet_bootstrap.cc:654] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.377055 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40dc4e83a19c408d9bc021089cb94c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.378239 2886 raft_consensus.cc:513] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.379283 2886 leader_election.cc:290] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.379405 1164 raft_consensus.cc:383] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.379501 1164 raft_consensus.cc:738] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.379745 1164 consensus_queue.cc:260] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.380290 1164 ts_tablet_manager.cc:1428] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20250112 20:51:50.380582 1164 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:50.386759 710 leader_election.cc:343] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.387761 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcf88720596d46628ef1fbd6737582d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.342545 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet b85b9e82f9e847c4acfeecb82201e694 with delete_type TABLET_DATA_DELETED (replaced by tablet e0bbf979d0b841899a2990a39d83394e) from {username='slave'} at 127.0.0.1:33760
W20250112 20:51:50.391628 710 leader_election.cc:343] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.399097 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_93db35038f744bfdbd5526c04bc025a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.399573 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93db35038f744bfdbd5526c04bc025a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.340704 2447 raft_consensus.cc:2267] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.400254 2447 ts_tablet_manager.cc:1905] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.400660 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90428f7ce05e4667a57dc6fec766a9be" } modify_external_catalogs: true
I20250112 20:51:50.400820 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_90428f7ce05e4667a57dc6fec766a9be" } modify_external_catalogs: true
W20250112 20:51:50.401059 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 443b29a93a0245e9a3a256e65a272699 consensus state has no leader
I20250112 20:51:50.404037 878 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b722b0eb5944a4b344a3833905a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:50.410095 710 leader_election.cc:343] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.411659 2632 raft_consensus.cc:491] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.411886 1164 tablet_bootstrap.cc:654] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:50.412113 710 leader_election.cc:343] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.413069 2632 raft_consensus.cc:513] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.422345 2886 consensus_queue.cc:1035] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250112 20:51:50.424325 3114 consensus_queue.cc:1035] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250112 20:51:50.424371 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_53510fe9a4124c20880d08d01510903c" } new_table_name: "default.Table_e79b993a193a463a9b6ab95f0e8e1b65" modify_external_catalogs: true
I20250112 20:51:50.424842 2632 leader_election.cc:290] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.424321 2923 consensus_queue.cc:1035] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250112 20:51:50.450443 2563 raft_consensus.cc:491] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.450636 2563 raft_consensus.cc:513] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.451303 2563 leader_election.cc:290] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.452293 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_8d9f63102f7546f9b40db699c7d49be9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.453003 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d9f63102f7546f9b40db699c7d49be9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.454527 2563 raft_consensus.cc:491] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.452318 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b3ec982cbf00412692d213f224f81413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:50.454651 2563 raft_consensus.cc:513] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:50.454939 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3ec982cbf00412692d213f224f81413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.455256 2563 leader_election.cc:290] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.457489 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.457710 906 raft_consensus.cc:2388] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:50.457649 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.457820 908 raft_consensus.cc:2463] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:50.458441 983 leader_election.cc:304] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:50.459760 1186 tablet_replica.cc:331] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.459750 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_86a42a16a5274fd4a667d4b7df06b8ea" } new_table_name: "default.Table_8d7496dac6374e5186e874f341b74977" modify_external_catalogs: true
I20250112 20:51:50.457669 1178 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:50.459980 1186 raft_consensus.cc:2238] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.460142 1186 raft_consensus.cc:2267] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.460680 1186 ts_tablet_manager.cc:1905] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.461406 1178 tablet_bootstrap.cc:654] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.463927 1163 tablet_bootstrap.cc:492] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:50.464047 1163 ts_tablet_manager.cc:1397] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.090s user 0.001s sys 0.001s
I20250112 20:51:50.464715 1163 raft_consensus.cc:357] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.465008 2563 raft_consensus.cc:2798] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:50.467985 1163 raft_consensus.cc:383] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.468472 1186 ts_tablet_manager.cc:1918] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.468001 2563 raft_consensus.cc:491] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.468621 1163 raft_consensus.cc:738] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.468653 2563 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:50.468896 1163 consensus_queue.cc:260] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.469374 1163 ts_tablet_manager.cc:1428] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:50.470934 2563 raft_consensus.cc:513] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.472355 2563 leader_election.cc:290] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.468575 1186 log.cc:1198] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f875f075e98e45499291b519caf03184
I20250112 20:51:50.472441 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.472679 908 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:50.472993 1186 ts_tablet_manager.cc:1939] T f875f075e98e45499291b519caf03184 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.473311 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 989) took 2262 ms. Trace:
I20250112 20:51:50.473393 1186 rpcz_store.cc:276] 0112 20:51:48.210570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.210611 (+ 41us) service_pool.cc:225] Handling call
0112 20:51:50.422346 (+2211735us) ts_tablet_manager.cc:1145] Deleting tablet f875f075e98e45499291b519caf03184
0112 20:51:50.459719 (+ 37373us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.464746 (+ 5027us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.468398 (+ 3652us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.473295 (+ 4897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211538}
I20250112 20:51:50.473681 1186 tablet_replica.cc:331] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.473847 1186 raft_consensus.cc:2238] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.473995 1186 raft_consensus.cc:2267] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.474476 1186 ts_tablet_manager.cc:1905] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.476087 908 raft_consensus.cc:2463] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 2.
I20250112 20:51:50.476552 983 leader_election.cc:304] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:50.476924 2563 raft_consensus.cc:2798] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:50.477088 2563 raft_consensus.cc:695] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:50.477360 2563 consensus_queue.cc:237] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.484620 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (table default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]) successfully deleted
I20250112 20:51:50.485984 1163 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:50.490952 1163 tablet_bootstrap.cc:654] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.498765 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:50.499863 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:50.500540 2563 raft_consensus.cc:491] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.500761 2563 raft_consensus.cc:513] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.502059 2563 leader_election.cc:290] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.502749 448 catalog_manager.cc:5526] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:50.503021 3258 sys_catalog.cc:263] Verifying existing consensus state
I20250112 20:51:50.503105 448 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:50.505373 1186 ts_tablet_manager.cc:1918] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:50.505492 1186 log.cc:1198] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f76dbd7f681b418ca7bb6c6e8f1f1e58
I20250112 20:51:50.505795 1186 ts_tablet_manager.cc:1939] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.506171 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 988) took 2295 ms. Trace:
I20250112 20:51:50.506260 1186 rpcz_store.cc:276] 0112 20:51:48.210490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.210965 (+ 475us) service_pool.cc:225] Handling call
0112 20:51:50.473638 (+2262673us) ts_tablet_manager.cc:1145] Deleting tablet f76dbd7f681b418ca7bb6c6e8f1f1e58
0112 20:51:50.473648 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.491992 (+ 18344us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.505277 (+ 13285us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.506153 (+ 876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262501}
I20250112 20:51:50.506671 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 990) took 2295 ms. Trace:
I20250112 20:51:50.506742 1186 rpcz_store.cc:276] 0112 20:51:48.210929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.211168 (+ 239us) service_pool.cc:225] Handling call
0112 20:51:50.506569 (+2295401us) ts_tablet_manager.cc:1145] Deleting tablet 4d29316b9d2441bca97f56bdb0e56f0d
0112 20:51:50.506580 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.506657 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2295265}
I20250112 20:51:50.506989 1186 tablet_replica.cc:331] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.507182 1185 ts_tablet_manager.cc:1918] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.507372 1185 log.cc:1198] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/226ea7f38f1445c598676715a3036227
I20250112 20:51:50.508945 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:50.510107 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.511337 1186 raft_consensus.cc:2238] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.511493 1186 raft_consensus.cc:2267] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.512045 1186 ts_tablet_manager.cc:1905] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.512233 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 5bb59c8a024e4d1faf0071df62633d60 (DEFAULT_TABLE table=default.Table_074b8d5f172f47cd856859c9b1e4c702 [id=06fd223b2b034cb886f64e034bfb0902]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.513937 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d3eba7aee3e441c1b668d9073b9b8e77 (DEFAULT_TABLE table=default.Table_04fff3dcd9d6433e85abc01fd00c77c1 [id=b382d64c6a83427faad45e68e9e6e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.514557 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 62796226707a4a768bcd699c1260d522 (DEFAULT_TABLE table=default.Table_8523098ce690434c99af052e3170ca62 [id=872e7380bf744485b6b8ef113e7d64f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.514752 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0c67644464da4b4ce3aaf7d1071c7 (DEFAULT_TABLE table=default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.515460 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 369c6454846b4615950b76abb0ef8d29 (DEFAULT_TABLE table=default.Table_3f7793320d1b43deb2bc7e107686169f [id=2ace54c8322947e7900431fe5fea281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.516316 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4a7a33e2dd7b42ada35718df1413bdd4" } modify_external_catalogs: true
I20250112 20:51:50.516516 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_4a7a33e2dd7b42ada35718df1413bdd4" } modify_external_catalogs: true
I20250112 20:51:50.519418 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5bb59c8a024e4d1faf0071df62633d60 (DEFAULT_TABLE table=default.Table_074b8d5f172f47cd856859c9b1e4c702 [id=06fd223b2b034cb886f64e034bfb0902]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.520260 1185 ts_tablet_manager.cc:1939] T 226ea7f38f1445c598676715a3036227 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:50.520403 756 tablet_service.cc:1467] Processing CreateTablet for tablet d3eba7aee3e441c1b668d9073b9b8e77 (DEFAULT_TABLE table=default.Table_04fff3dcd9d6433e85abc01fd00c77c1 [id=b382d64c6a83427faad45e68e9e6e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.520505 742 tablet_service.cc:1467] Processing CreateTablet for tablet 369c6454846b4615950b76abb0ef8d29 (DEFAULT_TABLE table=default.Table_3f7793320d1b43deb2bc7e107686169f [id=2ace54c8322947e7900431fe5fea281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.520419 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fdc1f15839ae4f91a0b63261f416283a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.520931 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdc1f15839ae4f91a0b63261f416283a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.522145 750 tablet_service.cc:1467] Processing CreateTablet for tablet 62796226707a4a768bcd699c1260d522 (DEFAULT_TABLE table=default.Table_8523098ce690434c99af052e3170ca62 [id=872e7380bf744485b6b8ef113e7d64f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.522755 754 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0c67644464da4b4ce3aaf7d1071c7 (DEFAULT_TABLE table=default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.522930 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb59c8a024e4d1faf0071df62633d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.534875 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c6454846b4615950b76abb0ef8d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.537433 1164 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:50.537667 1164 ts_tablet_manager.cc:1397] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.004s
I20250112 20:51:50.540970 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1b0c67644464da4b4ce3aaf7d1071c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.542102 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62796226707a4a768bcd699c1260d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.542984 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3eba7aee3e441c1b668d9073b9b8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.548854 3085 tablet.cc:1721] T 56544c3b0cf140089baa678765a2eb79 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:50.550086 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 128518us
I20250112 20:51:50.550523 3268 consensus_queue.cc:1035] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.550788 3097 tablet.cc:1721] T 56544c3b0cf140089baa678765a2eb79 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.551146 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 999) took 2545 ms. Trace:
I20250112 20:51:50.551312 1185 rpcz_store.cc:276] 0112 20:51:48.005674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.005723 (+ 49us) service_pool.cc:225] Handling call
0112 20:51:50.319754 (+2314031us) ts_tablet_manager.cc:1145] Deleting tablet 226ea7f38f1445c598676715a3036227
0112 20:51:50.319763 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.449826 (+130063us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.454577 (+ 4751us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.551107 (+ 96530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310666}
I20250112 20:51:50.551968 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1004) took 2535 ms. Trace:
I20250112 20:51:50.554884 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80791d8582446b89adf212d3d6507eb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.555092 758 raft_consensus.cc:2463] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:50.560882 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.561100 908 raft_consensus.cc:2463] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.561393 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.565120 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.567027 1039 raft_consensus.cc:3054] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.567719 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.567906 911 raft_consensus.cc:2463] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.584851 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f875f075e98e45499291b519caf03184 (table default.Table_0a39583e302f435bb0ae3f12b56e8e51 [id=a41262782d3c41bea2fc1decd8ba5a95]) successfully deleted
I20250112 20:51:50.585041 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 226ea7f38f1445c598676715a3036227 (table default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]) successfully deleted
I20250112 20:51:50.595042 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:50.599311 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb59c8a024e4d1faf0071df62633d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.599565 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62796226707a4a768bcd699c1260d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.599752 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3eba7aee3e441c1b668d9073b9b8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.600214 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1b0c67644464da4b4ce3aaf7d1071c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.600706 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c6454846b4615950b76abb0ef8d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.552115 1185 rpcz_store.cc:276] 0112 20:51:48.016060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.017392 (+ 1332us) service_pool.cc:225] Handling call
0112 20:51:50.551807 (+2534415us) ts_tablet_manager.cc:1145] Deleting tablet 146994989d6d4ce8a12e1b8677d7feb7
0112 20:51:50.551817 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.551937 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2534278}
I20250112 20:51:50.603351 1185 tablet_replica.cc:331] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:50.603665 1185 raft_consensus.cc:2238] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.603952 1185 raft_consensus.cc:2267] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.605057 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.605711 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.605962 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde64da3dd084a4785986c091852cbc6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.605700 1039 raft_consensus.cc:3054] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.606315 908 raft_consensus.cc:3054] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.609081 908 raft_consensus.cc:2463] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.609645 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480dca3c3a1e4da7a84834e02f26b299" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.609863 1046 raft_consensus.cc:3054] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.604959 1185 ts_tablet_manager.cc:1905] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.612627 1046 raft_consensus.cc:2463] T 480dca3c3a1e4da7a84834e02f26b299 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.613237 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.609817 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9570269368e14b1abeaffdcc741c6f3a" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.616159 905 raft_consensus.cc:3054] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.618733 905 raft_consensus.cc:2463] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.610913 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480dca3c3a1e4da7a84834e02f26b299" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.618986 912 raft_consensus.cc:3054] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.621356 912 raft_consensus.cc:2463] T 480dca3c3a1e4da7a84834e02f26b299 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.614331 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.628448 2447 ts_tablet_manager.cc:1918] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.631129 3097 tablet.cc:1721] T 97d32367b39a4cce8371377ec7449586 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:50.638268 3258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Bootstrap starting.
I20250112 20:51:50.641036 3097 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.646270 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.646772 761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca8fc4a45fe448b919ac001a7980deb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:50.648066 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76dbd7f681b418ca7bb6c6e8f1f1e58" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.648376 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85b9e82f9e847c4acfeecb82201e694" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.648562 771 raft_consensus.cc:2388] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:50.660076 3287 tablet.cc:1721] T dea98c29bc744009ab9b3eeb53684f29 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 20:51:50.662039 710 leader_election.cc:343] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:50.662408 710 leader_election.cc:343] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:50.662513 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18daa381181549228ad521157bf8793f consensus state has no leader
I20250112 20:51:50.662912 3144 consensus_queue.cc:1035] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:50.663792 710 leader_election.cc:343] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.664342 710 leader_election.cc:304] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.664604 3144 raft_consensus.cc:2798] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.664677 3144 raft_consensus.cc:491] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.664743 3144 raft_consensus.cc:3054] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.667068 1178 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:50.667212 1178 ts_tablet_manager.cc:1397] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.210s user 0.005s sys 0.007s
I20250112 20:51:50.667220 3144 raft_consensus.cc:513] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.667743 3144 leader_election.cc:290] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:50.668073 1178 raft_consensus.cc:357] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.668260 1178 raft_consensus.cc:383] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.668318 1178 raft_consensus.cc:738] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.668459 3144 consensus_queue.cc:1035] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.668511 1178 consensus_queue.cc:260] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.669222 1178 ts_tablet_manager.cc:1428] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:50.669525 1178 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:50.671295 1178 tablet_bootstrap.cc:654] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.683046 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b3f8eb107694ac5a7c7de6d5609352f" } modify_external_catalogs: true
I20250112 20:51:50.683252 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9b3f8eb107694ac5a7c7de6d5609352f" } modify_external_catalogs: true
I20250112 20:51:50.684377 1186 ts_tablet_manager.cc:1918] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:50.684489 1186 log.cc:1198] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/1ca8fc4a45fe448b919ac001a7980deb
W20250112 20:51:50.687379 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 46756.2us
I20250112 20:51:50.688372 1185 ts_tablet_manager.cc:1918] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:50.688491 1185 log.cc:1198] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/71743c73b6f24da9b14a3d4148117813
I20250112 20:51:50.688923 1185 ts_tablet_manager.cc:1939] T 71743c73b6f24da9b14a3d4148117813 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:50.689796 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1012) took 2576 ms. Trace:
I20250112 20:51:50.695469 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:50.694398 3144 consensus_queue.cc:1035] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.696524 710 leader_election.cc:304] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.707245 3144 raft_consensus.cc:2798] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.707341 3144 raft_consensus.cc:491] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.707424 3144 raft_consensus.cc:3054] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:50.707556 710 consensus_peers.cc:487] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f76dbd7f681b418ca7bb6c6e8f1f1e58. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:50.710242 3144 raft_consensus.cc:513] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.710991 3144 leader_election.cc:290] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.711206 1185 rpcz_store.cc:276] 0112 20:51:48.112954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.113172 (+ 218us) service_pool.cc:225] Handling call
0112 20:51:50.601558 (+2488386us) ts_tablet_manager.cc:1145] Deleting tablet 71743c73b6f24da9b14a3d4148117813
0112 20:51:50.603284 (+ 1726us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.639181 (+ 35897us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.688259 (+ 49078us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.689772 (+ 1513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2488190}
I20250112 20:51:50.711547 1186 ts_tablet_manager.cc:1939] T 1ca8fc4a45fe448b919ac001a7980deb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.712576 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 991) took 2501 ms. Trace:
I20250112 20:51:50.715235 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1013) took 2569 ms. Trace:
I20250112 20:51:50.715358 1185 rpcz_store.cc:276] 0112 20:51:48.145718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.145896 (+ 178us) service_pool.cc:225] Handling call
0112 20:51:50.712063 (+2566167us) ts_tablet_manager.cc:1145] Deleting tablet 2c8c7784e5534c2c91f124dffd24d5c5
0112 20:51:50.712077 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.715218 (+ 3141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2565968}
I20250112 20:51:50.710856 710 leader_election.cc:304] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.715729 1186 rpcz_store.cc:276] 0112 20:51:48.211398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.211434 (+ 36us) service_pool.cc:225] Handling call
0112 20:51:50.506945 (+2295511us) ts_tablet_manager.cc:1145] Deleting tablet 1ca8fc4a45fe448b919ac001a7980deb
0112 20:51:50.506951 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.681205 (+174254us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.684278 (+ 3073us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.712559 (+ 28281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2295371}
I20250112 20:51:50.715935 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (table default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]) successfully deleted
I20250112 20:51:50.716248 3144 raft_consensus.cc:2798] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:50.716265 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4d29316b9d2441bca97f56bdb0e56f0d because the tablet was not found. No further retry: Not found: Tablet not found: 4d29316b9d2441bca97f56bdb0e56f0d
I20250112 20:51:50.716492 3144 raft_consensus.cc:695] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.716748 3144 consensus_queue.cc:237] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:50.717774 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 146994989d6d4ce8a12e1b8677d7feb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146994989d6d4ce8a12e1b8677d7feb7 already in progress: creating tablet
I20250112 20:51:50.719527 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1014) took 2491 ms. Trace:
I20250112 20:51:50.719691 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]) successfully deleted
I20250112 20:51:50.719669 1185 rpcz_store.cc:276] 0112 20:51:48.227989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.229861 (+ 1872us) service_pool.cc:225] Handling call
0112 20:51:50.719418 (+2489557us) ts_tablet_manager.cc:1145] Deleting tablet cfb833659c23431daaab008dcdf1de43
0112 20:51:50.719430 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.719510 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406549}
I20250112 20:51:50.720275 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1016) took 2432 ms. Trace:
I20250112 20:51:50.720408 1185 rpcz_store.cc:276] 0112 20:51:48.288141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.289458 (+ 1317us) service_pool.cc:225] Handling call
0112 20:51:50.720160 (+2430702us) ts_tablet_manager.cc:1145] Deleting tablet db9bab2477a94c169817d9de8aaad1c0
0112 20:51:50.720171 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.720258 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407099}
I20250112 20:51:50.720959 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1023) took 2371 ms. Trace:
I20250112 20:51:50.716145 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 992) took 2504 ms. Trace:
I20250112 20:51:50.721094 1185 rpcz_store.cc:276] 0112 20:51:48.349527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.349587 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:50.720868 (+2371281us) ts_tablet_manager.cc:1145] Deleting tablet b4ef01b41b6140d6bdad07bc79dd7dc5
0112 20:51:50.720878 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.720943 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2371002}
I20250112 20:51:50.721199 1186 rpcz_store.cc:276] 0112 20:51:48.211675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.211712 (+ 37us) service_pool.cc:225] Handling call
0112 20:51:50.716027 (+2504315us) ts_tablet_manager.cc:1145] Deleting tablet 0bb9c2e6f37f4184a106aed05bae7924
0112 20:51:50.716038 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.716129 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2504152}
I20250112 20:51:50.721623 1185 tablet_replica.cc:331] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:50.721689 710 leader_election.cc:304] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.721817 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 993) took 2509 ms. Trace:
I20250112 20:51:50.721870 1185 raft_consensus.cc:2238] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.722030 1185 raft_consensus.cc:2267] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.721951 1186 rpcz_store.cc:276] 0112 20:51:48.211972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.216845 (+ 4873us) service_pool.cc:225] Handling call
0112 20:51:50.721610 (+2504765us) ts_tablet_manager.cc:1145] Deleting tablet 9705feb15f384b5185935eacc521d0e0
0112 20:51:50.721620 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.721778 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2504570}
I20250112 20:51:50.722244 3268 raft_consensus.cc:2798] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.722376 3268 raft_consensus.cc:695] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.722502 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 994) took 2505 ms. Trace:
I20250112 20:51:50.722641 3268 consensus_queue.cc:237] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.722635 1186 rpcz_store.cc:276] 0112 20:51:48.217170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.217209 (+ 39us) service_pool.cc:225] Handling call
0112 20:51:50.722341 (+2505132us) ts_tablet_manager.cc:1145] Deleting tablet 7d9572179a6f45759214c6c39f22b2f0
0112 20:51:50.722351 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.722486 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2504936}
I20250112 20:51:50.723255 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 995) took 2505 ms. Trace:
I20250112 20:51:50.723392 1186 rpcz_store.cc:276] 0112 20:51:48.217492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.217535 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:50.723030 (+2505495us) ts_tablet_manager.cc:1145] Deleting tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8
0112 20:51:50.723070 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.723214 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505328}
I20250112 20:51:50.724215 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 996) took 2506 ms. Trace:
I20250112 20:51:50.724311 1186 rpcz_store.cc:276] 0112 20:51:48.217804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.217842 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:50.724039 (+2506197us) ts_tablet_manager.cc:1145] Deleting tablet 438e2c14353c4a8cb7242d75bb3ba673
0112 20:51:50.724048 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.724174 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506017}
I20250112 20:51:50.725057 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 997) took 2506 ms. Trace:
I20250112 20:51:50.725198 1186 rpcz_store.cc:276] 0112 20:51:48.218135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.218173 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:50.724686 (+2506513us) ts_tablet_manager.cc:1145] Deleting tablet b062c1d67d4443eabe8ac1d8a832c243
0112 20:51:50.724695 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.725016 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506320}
I20250112 20:51:50.725549 1186 tablet_replica.cc:331] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.725878 1186 raft_consensus.cc:2238] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.726104 1186 raft_consensus.cc:2267] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.727007 1186 ts_tablet_manager.cc:1905] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.730854 1185 ts_tablet_manager.cc:1905] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:50.734319 710 leader_election.cc:343] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.735124 710 leader_election.cc:304] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.737875 1186 ts_tablet_manager.cc:1918] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.738071 1186 log.cc:1198] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/226ea7f38f1445c598676715a3036227
I20250112 20:51:50.738757 1186 ts_tablet_manager.cc:1939] T 226ea7f38f1445c598676715a3036227 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.739455 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 998) took 2520 ms. Trace:
I20250112 20:51:50.740439 3121 raft_consensus.cc:2798] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.740562 3121 raft_consensus.cc:695] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.740767 3121 consensus_queue.cc:237] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.739595 1186 rpcz_store.cc:276] 0112 20:51:48.218451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.218510 (+ 59us) service_pool.cc:225] Handling call
0112 20:51:50.725484 (+2506974us) ts_tablet_manager.cc:1145] Deleting tablet 226ea7f38f1445c598676715a3036227
0112 20:51:50.725492 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.733004 (+ 7512us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.737699 (+ 4695us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.739303 (+ 1604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506791}
I20250112 20:51:50.743533 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1000) took 2524 ms. Trace:
I20250112 20:51:50.743665 1186 rpcz_store.cc:276] 0112 20:51:48.219224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.219270 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:50.743344 (+2524074us) ts_tablet_manager.cc:1145] Deleting tablet 6b440da020d947dda2553d65dd6a4bdf
0112 20:51:50.743356 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.743516 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2523870}
I20250112 20:51:50.744499 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1005) took 2475 ms. Trace:
I20250112 20:51:50.744609 1186 rpcz_store.cc:276] 0112 20:51:48.269352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.273151 (+ 3799us) service_pool.cc:225] Handling call
0112 20:51:50.744346 (+2471195us) ts_tablet_manager.cc:1145] Deleting tablet 146994989d6d4ce8a12e1b8677d7feb7
0112 20:51:50.744356 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.744483 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470571}
I20250112 20:51:50.744944 1186 tablet_replica.cc:331] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.745242 1186 raft_consensus.cc:2238] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.745626 1186 raft_consensus.cc:2267] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.746631 1186 ts_tablet_manager.cc:1905] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.631108 1039 raft_consensus.cc:2463] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.649140 2563 raft_consensus.cc:491] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.765107 2563 raft_consensus.cc:513] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.765846 2563 leader_election.cc:290] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:50.661198 983 consensus_peers.cc:487] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fd34e829d0645ea80daaff5d27cb266. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:50.766273 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 105110us
I20250112 20:51:50.693039 1044 raft_consensus.cc:1235] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 LEADER]: Rejecting Update request from peer aefe1db8b19244d891b6bcd8971e9b17 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250112 20:51:50.719921 1163 tablet_bootstrap.cc:492] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:50.768713 1163 ts_tablet_manager.cc:1397] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.283s user 0.016s sys 0.023s
I20250112 20:51:50.769487 1163 raft_consensus.cc:357] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.769724 1163 raft_consensus.cc:383] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.769794 1163 raft_consensus.cc:738] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.770709 1163 consensus_queue.cc:260] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.771199 1163 ts_tablet_manager.cc:1428] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:50.771999 1163 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:50.773733 1163 tablet_bootstrap.cc:654] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.720904 3284 raft_consensus.cc:491] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.775090 3284 raft_consensus.cc:513] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:50.776017 710 consensus_peers.cc:487] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:50.776888 3284 leader_election.cc:290] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.779832 2447 log.cc:1198] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1d97ac93467a4a7a8ae44d52719f191c
I20250112 20:51:50.752590 1047 raft_consensus.cc:3054] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.753628 3085 tablet.cc:1721] T 97d32367b39a4cce8371377ec7449586 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:50.754990 3250 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.781250 3287 tablet.cc:1721] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:50.781658 2447 ts_tablet_manager.cc:1939] T 1d97ac93467a4a7a8ae44d52719f191c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:50.785215 1186 ts_tablet_manager.cc:1918] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:50.786162 1186 log.cc:1198] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/71743c73b6f24da9b14a3d4148117813
I20250112 20:51:50.754432 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9db0cdd2cd10494c9bff7b59017806a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.787526 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db0cdd2cd10494c9bff7b59017806a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:50.758971 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05550152671e4e28829b5b102b59f715 consensus state has no leader
I20250112 20:51:50.789029 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_44d406e2cfe24b0db868598432e2a504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:50.789317 1186 ts_tablet_manager.cc:1939] T 71743c73b6f24da9b14a3d4148117813 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:50.789498 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d406e2cfe24b0db868598432e2a504 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.789602 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1013) took 2412 ms. Trace:
I20250112 20:51:50.789678 1186 rpcz_store.cc:276] 0112 20:51:48.376915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.377069 (+ 154us) service_pool.cc:225] Handling call
0112 20:51:50.744883 (+2367814us) ts_tablet_manager.cc:1145] Deleting tablet 71743c73b6f24da9b14a3d4148117813
0112 20:51:50.744891 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.751615 (+ 6724us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.784608 (+ 32993us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.789589 (+ 4981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367533}
I20250112 20:51:50.790038 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1014) took 2413 ms. Trace:
I20250112 20:51:50.790107 1186 rpcz_store.cc:276] 0112 20:51:48.377029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.384025 (+ 6996us) service_pool.cc:225] Handling call
0112 20:51:50.789930 (+2405905us) ts_tablet_manager.cc:1145] Deleting tablet 2c8c7784e5534c2c91f124dffd24d5c5
0112 20:51:50.789939 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.790027 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405687}
I20250112 20:51:50.790369 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1015) took 2405 ms. Trace:
I20250112 20:51:50.790421 1186 rpcz_store.cc:276] 0112 20:51:48.384973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.401622 (+ 16649us) service_pool.cc:225] Handling call
0112 20:51:50.790289 (+2388667us) ts_tablet_manager.cc:1145] Deleting tablet cfb833659c23431daaab008dcdf1de43
0112 20:51:50.790296 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.790357 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352593}
I20250112 20:51:50.790601 1186 tablet_replica.cc:331] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.790761 1186 raft_consensus.cc:2238] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:50.790907 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 71212.9us
W20250112 20:51:50.792685 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 2c8c7784e5534c2c91f124dffd24d5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8c7784e5534c2c91f124dffd24d5c5 already in progress: creating tablet
I20250112 20:51:50.793121 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 968) took 2391 ms. Trace:
W20250112 20:51:50.794512 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cfb833659c23431daaab008dcdf1de43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb833659c23431daaab008dcdf1de43 already in progress: creating tablet
I20250112 20:51:50.796123 2447 rpcz_store.cc:276] 0112 20:51:48.401214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.401259 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:50.339364 (+1938105us) ts_tablet_manager.cc:1145] Deleting tablet 1d97ac93467a4a7a8ae44d52719f191c
0112 20:51:50.339402 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.625448 (+286046us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.628352 (+ 2904us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.793100 (+164748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929882}
I20250112 20:51:50.797343 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.797610 911 raft_consensus.cc:3054] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.798817 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:50.798979 910 raft_consensus.cc:3054] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:50.801081 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet db9bab2477a94c169817d9de8aaad1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9bab2477a94c169817d9de8aaad1c0 already in progress: creating tablet
I20250112 20:51:50.801223 910 raft_consensus.cc:2463] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.797089 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.798563 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:50.803860 2447 tablet_replica.cc:331] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:50.804092 2447 raft_consensus.cc:2238] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.802671 983 leader_election.cc:304] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:50.804603 2787 raft_consensus.cc:2743] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.804607 3268 raft_consensus.cc:491] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.804744 3268 raft_consensus.cc:513] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.805114 983 leader_election.cc:304] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:50.805320 2787 raft_consensus.cc:2743] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.805619 3268 leader_election.cc:290] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.807791 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b74aaf7f5b4c87acbefaa4df5c10d2" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:50.810627 710 leader_election.cc:343] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.811092 710 leader_election.cc:304] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:50.811779 3268 raft_consensus.cc:2743] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.812690 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 886690f24fa540a6b496c926bc7a2b07
W20250112 20:51:50.829447 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b4ef01b41b6140d6bdad07bc79dd7dc5 because the tablet was not found. No further retry: Not found: Tablet not found: b4ef01b41b6140d6bdad07bc79dd7dc5
I20250112 20:51:50.821605 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 056ed0fd01824bcea53d58251328898c
I20250112 20:51:50.831497 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 886690f24fa540a6b496c926bc7a2b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.831487 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 886690f24fa540a6b496c926bc7a2b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.832391 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1d97ac93467a4a7a8ae44d52719f191c (table default.Table_5b2a82b2202e4bc6b5164fe9c4f07829 [id=4260d5d23b454b6ea05a302c28e2e4d4]) successfully deleted
I20250112 20:51:50.833019 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 056ed0fd01824bcea53d58251328898c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.833012 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 056ed0fd01824bcea53d58251328898c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.839819 3287 tablet.cc:1721] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.841581 911 raft_consensus.cc:2463] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:50.846226 3144 consensus_queue.cc:1046] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.846496 3144 consensus_queue.cc:1225] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Peer 30ba399c7a6749e0acd075d93e4cc4b3 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 20:51:50.846616 2447 raft_consensus.cc:2267] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.846616 2886 raft_consensus.cc:3049] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 20:51:50.846698 2886 raft_consensus.cc:738] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.846933 2886 consensus_queue.cc:260] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.847272 2886 raft_consensus.cc:3054] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:50.847831 1163 tablet_bootstrap.cc:492] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:50.847972 1163 ts_tablet_manager.cc:1397] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.040s
I20250112 20:51:50.848572 1163 raft_consensus.cc:357] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.848757 1163 raft_consensus.cc:383] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.848840 1163 raft_consensus.cc:738] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.849033 1163 consensus_queue.cc:260] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.849524 1163 ts_tablet_manager.cc:1428] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250112 20:51:50.849757 1163 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:50.851449 1163 tablet_bootstrap.cc:654] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.853080 847 leader_election.cc:304] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.853411 1186 pending_rounds.cc:62] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:50.853470 1186 pending_rounds.cc:69] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585088957857792 op_type: NO_OP noop_request { }
I20250112 20:51:50.853572 1186 raft_consensus.cc:2883] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:50.853648 1186 raft_consensus.cc:2267] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.853972 904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.854162 904 raft_consensus.cc:2388] T 0946fe64150e41da957907aa4a6a672f P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:50.854211 1186 ts_tablet_manager.cc:1905] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.854286 2521 raft_consensus.cc:2798] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.854377 2521 raft_consensus.cc:491] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.854437 2521 raft_consensus.cc:3054] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.854207 2447 ts_tablet_manager.cc:1905] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.858979 2521 raft_consensus.cc:513] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.859522 2521 leader_election.cc:290] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:50.861291 1186 ts_tablet_manager.cc:1918] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.861389 1186 log.cc:1198] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:50.861727 1186 ts_tablet_manager.cc:1939] T b85b9e82f9e847c4acfeecb82201e694 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.861986 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1017) took 2442 ms. Trace:
I20250112 20:51:50.862056 1186 rpcz_store.cc:276] 0112 20:51:48.419263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.419314 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:50.790569 (+2371255us) ts_tablet_manager.cc:1145] Deleting tablet b85b9e82f9e847c4acfeecb82201e694
0112 20:51:50.790575 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.858031 (+ 67456us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.861207 (+ 3176us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.861974 (+ 767us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62472,"tablet-delete.queue_time_us":2352804}
I20250112 20:51:50.862576 1186 tablet_replica.cc:331] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.862754 1186 raft_consensus.cc:2238] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:50.862890 1186 raft_consensus.cc:2267] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.863386 1186 ts_tablet_manager.cc:1905] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:50.866667 983 leader_election.cc:343] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 61b74aaf7f5b4c87acbefaa4df5c10d2
W20250112 20:51:50.874352 710 leader_election.cc:343] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.875036 1178 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:50.874958 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_72b8c68194b84004b32fa74e32d5e26e" } new_table_name: "default.Table_89b51b6ef2d74df7a45fb8c00b3b4538" modify_external_catalogs: true
I20250112 20:51:50.875456 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_de5c34db50e64848a3fb7a3d1a4e3613" } new_table_name: "default.Table_8198a5e6e8db4607b1a0604759f5c009" modify_external_catalogs: true
I20250112 20:51:50.875764 1178 ts_tablet_manager.cc:1397] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.206s user 0.002s sys 0.002s
I20250112 20:51:50.877099 1178 raft_consensus.cc:357] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:50.877893 710 leader_election.cc:343] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:50.879112 452 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:50.880610 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:50.882185 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 43d341f6ac6848d9a8e60f689dd24a66 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:50.888762 1185 ts_tablet_manager.cc:1918] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.888917 1185 log.cc:1198] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/49b8cc99742f4f9ba68eb1aa17cab248
I20250112 20:51:50.889279 710 leader_election.cc:304] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.889968 3265 raft_consensus.cc:2798] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.890072 3265 raft_consensus.cc:695] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.890312 3265 consensus_queue.cc:237] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.890882 1164 raft_consensus.cc:357] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.891041 1164 raft_consensus.cc:383] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.891100 1164 raft_consensus.cc:738] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.891305 1164 consensus_queue.cc:260] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.891840 1164 ts_tablet_manager.cc:1428] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:50.892246 1164 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:50.895206 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a00f71eb25f84488a78ed3bea61f57e7" } modify_external_catalogs: true
I20250112 20:51:50.895437 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a00f71eb25f84488a78ed3bea61f57e7" } modify_external_catalogs: true
I20250112 20:51:50.897646 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_01d0b758a621413cb1fb3e51347fde91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.898291 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01d0b758a621413cb1fb3e51347fde91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.904979 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 096ee7d3a66846f9b619d8f2fc046616
I20250112 20:51:50.884622 1186 ts_tablet_manager.cc:1918] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.906519 1186 log.cc:1198] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e0fd4177f9394a5a82d4f5e02369b3b6
I20250112 20:51:50.906992 1186 ts_tablet_manager.cc:1939] T e0fd4177f9394a5a82d4f5e02369b3b6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.906553 1178 raft_consensus.cc:383] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.907130 3097 tablet.cc:1721] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:50.907405 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1018) took 2487 ms. Trace:
I20250112 20:51:50.907487 1186 rpcz_store.cc:276] 0112 20:51:48.419667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.419879 (+ 212us) service_pool.cc:225] Handling call
0112 20:51:50.862310 (+2442431us) ts_tablet_manager.cc:1145] Deleting tablet e0fd4177f9394a5a82d4f5e02369b3b6
0112 20:51:50.862537 (+ 227us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.866512 (+ 3975us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.884507 (+ 17995us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.907392 (+ 22885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419264}
I20250112 20:51:50.907931 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1019) took 2487 ms. Trace:
I20250112 20:51:50.908007 1186 rpcz_store.cc:276] 0112 20:51:48.420126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.420172 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:50.907753 (+2487581us) ts_tablet_manager.cc:1145] Deleting tablet db9bab2477a94c169817d9de8aaad1c0
0112 20:51:50.907769 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.907916 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464623}
I20250112 20:51:50.909544 3287 tablet.cc:1721] T e72a6c8e18934b789a048d5aaded121b P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.912941 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1021) took 2492 ms. Trace:
I20250112 20:51:50.913020 1186 rpcz_store.cc:276] 0112 20:51:48.420841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.420881 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:50.908195 (+2487314us) ts_tablet_manager.cc:1145] Deleting tablet 3ea9425f3e7b46288c81dc8cdade99b2
0112 20:51:50.908203 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.912924 (+ 4721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465015}
I20250112 20:51:50.913266 1186 tablet_replica.cc:331] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.913440 1186 raft_consensus.cc:2238] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.913686 1186 raft_consensus.cc:2267] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:50.914034 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 60394.4us
I20250112 20:51:50.914451 1186 ts_tablet_manager.cc:1905] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.914604 1178 raft_consensus.cc:738] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:50.914902 710 leader_election.cc:304] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:50.915534 3268 raft_consensus.cc:2798] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:50.915697 3268 raft_consensus.cc:695] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:50.915957 3268 consensus_queue.cc:237] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.918551 3178 consensus_queue.cc:1035] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:50.920637 875 tablet_service.cc:1467] Processing CreateTablet for tablet 369c6454846b4615950b76abb0ef8d29 (DEFAULT_TABLE table=default.Table_3f7793320d1b43deb2bc7e107686169f [id=2ace54c8322947e7900431fe5fea281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.921293 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5bb59c8a024e4d1faf0071df62633d60 (DEFAULT_TABLE table=default.Table_074b8d5f172f47cd856859c9b1e4c702 [id=06fd223b2b034cb886f64e034bfb0902]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.921474 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c6454846b4615950b76abb0ef8d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.921628 1186 ts_tablet_manager.cc:1918] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:50.921707 1186 log.cc:1198] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9989f3dc61df4f75b9bdb85ba3ca3f37
I20250112 20:51:50.921736 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb59c8a024e4d1faf0071df62633d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.922048 1186 ts_tablet_manager.cc:1939] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.922317 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1016) took 2520 ms. Trace:
I20250112 20:51:50.922384 1186 rpcz_store.cc:276] 0112 20:51:48.419100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.419730 (+ 630us) service_pool.cc:225] Handling call
0112 20:51:50.913226 (+2493496us) ts_tablet_manager.cc:1145] Deleting tablet 9989f3dc61df4f75b9bdb85ba3ca3f37
0112 20:51:50.913235 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.918950 (+ 5715us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.921563 (+ 2613us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.922304 (+ 741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470004}
I20250112 20:51:50.922624 1186 tablet_replica.cc:331] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.922771 1186 raft_consensus.cc:2238] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.923002 1186 raft_consensus.cc:2267] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.923453 1186 ts_tablet_manager.cc:1905] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.924430 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.924878 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.926023 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:50.927079 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_91202e68552445aeab8e24f4978d8203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.927670 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91202e68552445aeab8e24f4978d8203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.928879 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:50.930168 1178 consensus_queue.cc:260] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.930814 1178 ts_tablet_manager.cc:1428] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20250112 20:51:50.933578 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17e6d263014448b96c5d1f03013e6e7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:50.889281 1185 ts_tablet_manager.cc:1939] T 49b8cc99742f4f9ba68eb1aa17cab248 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:50.936589 1178 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:50.937579 983 leader_election.cc:304] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:50.938061 3297 raft_consensus.cc:2743] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.938136 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1022) took 2588 ms. Trace:
I20250112 20:51:50.939246 1186 ts_tablet_manager.cc:1918] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:50.941035 1041 raft_consensus.cc:3054] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.938972 1185 rpcz_store.cc:276] 0112 20:51:48.349432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.349988 (+ 556us) service_pool.cc:225] Handling call
0112 20:51:50.721580 (+2371592us) ts_tablet_manager.cc:1145] Deleting tablet 49b8cc99742f4f9ba68eb1aa17cab248
0112 20:51:50.721590 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.778577 (+ 56987us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.888664 (+110087us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.938096 (+ 49432us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6930,"tablet-delete.queue_time_us":2371427}
I20250112 20:51:50.946372 1185 tablet_replica.cc:331] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:50.946620 1185 raft_consensus.cc:2238] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.946986 1185 raft_consensus.cc:2267] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.947665 3250 tablet.cc:1721] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 20:51:50.948243 3085 tablet.cc:1721] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.956574 1185 ts_tablet_manager.cc:1905] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.958643 3297 raft_consensus.cc:491] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:50.958782 3297 raft_consensus.cc:513] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:50.959365 3297 leader_election.cc:290] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.944655 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17e6d263014448b96c5d1f03013e6e7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.961500 1186 log.cc:1198] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/45484f8e48524b86a6f418b88a71ac18
I20250112 20:51:50.961529 908 raft_consensus.cc:2463] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:50.945943 3097 tablet.cc:1721] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.961894 1186 ts_tablet_manager.cc:1939] T 45484f8e48524b86a6f418b88a71ac18 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:50.947472 889 tablet_service.cc:1467] Processing CreateTablet for tablet d3eba7aee3e441c1b668d9073b9b8e77 (DEFAULT_TABLE table=default.Table_04fff3dcd9d6433e85abc01fd00c77c1 [id=b382d64c6a83427faad45e68e9e6e595]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.962436 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3eba7aee3e441c1b668d9073b9b8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.962591 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1022) took 2053 ms. Trace:
I20250112 20:51:50.962668 1186 rpcz_store.cc:276] 0112 20:51:48.909371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.909491 (+ 120us) service_pool.cc:225] Handling call
0112 20:51:50.922589 (+2013098us) ts_tablet_manager.cc:1145] Deleting tablet 45484f8e48524b86a6f418b88a71ac18
0112 20:51:50.922597 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.934761 (+ 12164us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.939166 (+ 4405us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:50.962576 (+ 23410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012912}
I20250112 20:51:50.963032 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1024) took 2024 ms. Trace:
I20250112 20:51:50.963104 1186 rpcz_store.cc:276] 0112 20:51:48.938872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.938923 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:50.962918 (+2023995us) ts_tablet_manager.cc:1145] Deleting tablet dfc63eff44414906a027a8208b5cc5a9
0112 20:51:50.962926 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.963016 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2023769}
I20250112 20:51:50.963433 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1026) took 2024 ms. Trace:
I20250112 20:51:50.963497 1186 rpcz_store.cc:276] 0112 20:51:48.939094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.939206 (+ 112us) service_pool.cc:225] Handling call
0112 20:51:50.963321 (+2024115us) ts_tablet_manager.cc:1145] Deleting tablet ed925a5ef3a54f819ca292a4e7967b33
0112 20:51:50.963330 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.963418 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2023945}
I20250112 20:51:50.963717 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1027) took 2024 ms. Trace:
I20250112 20:51:50.963766 1186 rpcz_store.cc:276] 0112 20:51:48.939191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.939407 (+ 216us) service_pool.cc:225] Handling call
0112 20:51:50.963659 (+2024252us) ts_tablet_manager.cc:1145] Deleting tablet cbc22207e9204b73877c74f61ebba8a8
0112 20:51:50.963666 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.963706 (+ 40us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024135}
I20250112 20:51:50.964004 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1028) took 2024 ms. Trace:
I20250112 20:51:50.964054 1186 rpcz_store.cc:276] 0112 20:51:48.939286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.940227 (+ 941us) service_pool.cc:225] Handling call
0112 20:51:50.963925 (+2023698us) ts_tablet_manager.cc:1145] Deleting tablet 3846863c74f64923bda6e5a19e7f2770
0112 20:51:50.963931 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.963993 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2023577}
I20250112 20:51:50.964015 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_33f87eb1d7cf4f4a8b8e04ab31163bd3" } new_table_name: "default.Table_ac8ac3d56f6a4fab91d8076721a80955" modify_external_catalogs: true
I20250112 20:51:50.964254 1186 tablet_replica.cc:331] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:50.964406 1186 raft_consensus.cc:2238] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:50.964533 1186 pending_rounds.cc:62] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:50.964565 1186 pending_rounds.cc:69] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585090290475008 op_type: NO_OP noop_request { }
I20250112 20:51:50.964640 1186 raft_consensus.cc:2883] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:50.964701 1186 raft_consensus.cc:2267] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:50.967067 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0946fe64150e41da957907aa4a6a672f" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.968269 3287 tablet.cc:1721] T 56544c3b0cf140089baa678765a2eb79 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.947788 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.948640 884 tablet_service.cc:1467] Processing CreateTablet for tablet 62796226707a4a768bcd699c1260d522 (DEFAULT_TABLE table=default.Table_8523098ce690434c99af052e3170ca62 [id=872e7380bf744485b6b8ef113e7d64f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.971251 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62796226707a4a768bcd699c1260d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.967782 1178 tablet_bootstrap.cc:654] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.949146 891 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0c67644464da4b4ce3aaf7d1071c7 (DEFAULT_TABLE table=default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:50.976389 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:50.976619 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 8074b00eda8d434faaff4b98ae923ebe
I20250112 20:51:50.976667 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1b0c67644464da4b4ce3aaf7d1071c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.960392 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6b2c4ea5f4167896c78b1c64cbb62" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:50.979418 907 raft_consensus.cc:2388] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:50.983767 3097 tablet.cc:1721] T e72a6c8e18934b789a048d5aaded121b P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:50.983938 983 leader_election.cc:304] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:50.984362 3130 raft_consensus.cc:3054] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.985705 983 leader_election.cc:304] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:50.986083 3297 raft_consensus.cc:3054] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.987243 3130 raft_consensus.cc:2743] T 0946fe64150e41da957907aa4a6a672f P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.988687 3297 raft_consensus.cc:2743] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:50.988757 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d019aad7635645ecbede54659d8e4f91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:50.989413 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d019aad7635645ecbede54659d8e4f91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:50.989617 1186 ts_tablet_manager.cc:1905] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:50.991811 710 leader_election.cc:304] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:50.992333 3268 raft_consensus.cc:2798] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:50.992461 3268 raft_consensus.cc:491] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:50.992583 3268 raft_consensus.cc:3054] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:50.995313 3268 raft_consensus.cc:513] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:50.995500 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 886690f24fa540a6b496c926bc7a2b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.995995 3268 leader_election.cc:290] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:50.996042 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 056ed0fd01824bcea53d58251328898c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.996397 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 096ee7d3a66846f9b619d8f2fc046616 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:50.997572 3287 tablet.cc:1716] T 1e584b761d234c5aa0ab875acd8e1e05 P aefe1db8b19244d891b6bcd8971e9b17: Skipping requested alter to schema version 4, tablet already version 4
W20250112 20:51:51.001845 710 leader_election.cc:343] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.003454 3268 consensus_queue.cc:1035] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.004168 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8074b00eda8d434faaff4b98ae923ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.004704 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8074b00eda8d434faaff4b98ae923ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.005573 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 1ca8fc4a45fe448b919ac001a7980deb (table default.Table_a1454445414a4f2882deedb81bad67bf [id=a40bd70a54ec4854a36809ac58df29b0]) successfully deleted
W20250112 20:51:51.006067 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 0bb9c2e6f37f4184a106aed05bae7924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bb9c2e6f37f4184a106aed05bae7924 already in progress: creating tablet
I20250112 20:51:51.006369 1185 ts_tablet_manager.cc:1918] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:51.006369 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9705feb15f384b5185935eacc521d0e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9705feb15f384b5185935eacc521d0e0 already in progress: creating tablet
I20250112 20:51:51.006507 1185 log.cc:1198] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/669e1cfd407e4d87a63e6832d9c6fa1f
W20250112 20:51:51.006757 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 7d9572179a6f45759214c6c39f22b2f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9572179a6f45759214c6c39f22b2f0 already in progress: creating tablet
I20250112 20:51:51.006882 1185 ts_tablet_manager.cc:1939] T 669e1cfd407e4d87a63e6832d9c6fa1f P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
W20250112 20:51:51.007092 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1332cfdeaba454c8e0d9f4f0ceb1fc8 already in progress: creating tablet
W20250112 20:51:51.007488 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 438e2c14353c4a8cb7242d75bb3ba673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438e2c14353c4a8cb7242d75bb3ba673 already in progress: creating tablet
W20250112 20:51:51.007764 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b062c1d67d4443eabe8ac1d8a832c243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b062c1d67d4443eabe8ac1d8a832c243 already in progress: creating tablet
I20250112 20:51:51.010177 1044 raft_consensus.cc:3054] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.010561 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17e6d263014448b96c5d1f03013e6e7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:51.010939 1186 ts_tablet_manager.cc:1918] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.011054 1186 log.cc:1198] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:51.011402 1186 ts_tablet_manager.cc:1939] T 9880e7d303b54d959bca4e8464a6b30c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.012692 3250 tablet.cc:1721] T e72a6c8e18934b789a048d5aaded121b P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:51.015698 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80791d8582446b89adf212d3d6507eb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:51.017253 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1047) took 1681 ms. Trace:
I20250112 20:51:51.017357 1186 rpcz_store.cc:276] 0112 20:51:49.335321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.338308 (+ 2987us) service_pool.cc:225] Handling call
0112 20:51:50.964222 (+1625914us) ts_tablet_manager.cc:1145] Deleting tablet 9880e7d303b54d959bca4e8464a6b30c
0112 20:51:50.964229 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.992204 (+ 27975us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.010829 (+ 18625us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.017238 (+ 6409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625708}
I20250112 20:51:51.017652 1186 tablet_replica.cc:331] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.017765 3097 tablet.cc:1716] T 1e584b761d234c5aa0ab875acd8e1e05 P 87ae45957c64469d96dc92d606b5b4b3: Skipping requested alter to schema version 4, tablet already version 4
I20250112 20:51:51.017845 1186 raft_consensus.cc:2238] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.018003 1186 raft_consensus.cc:2267] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.017578 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17e6d263014448b96c5d1f03013e6e7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:51.018424 906 raft_consensus.cc:3054] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.018688 3250 tablet.cc:1716] T 1e584b761d234c5aa0ab875acd8e1e05 P 30ba399c7a6749e0acd075d93e4cc4b3: Skipping requested alter to schema version 4, tablet already version 4
I20250112 20:51:51.018452 1186 ts_tablet_manager.cc:1905] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.021248 906 raft_consensus.cc:2463] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:51.023530 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6a80d9a8edfd459aad1ba74017ec03ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.024300 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a80d9a8edfd459aad1ba74017ec03ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.026237 1186 ts_tablet_manager.cc:1918] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:51.026335 1186 log.cc:1198] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/9570269368e14b1abeaffdcc741c6f3a
I20250112 20:51:51.026620 1186 ts_tablet_manager.cc:1939] T 9570269368e14b1abeaffdcc741c6f3a P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.026913 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1021) took 2677 ms. Trace:
I20250112 20:51:51.028736 1185 rpcz_store.cc:276] 0112 20:51:48.349345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.352964 (+ 3619us) service_pool.cc:225] Handling call
0112 20:51:50.941811 (+2588847us) ts_tablet_manager.cc:1145] Deleting tablet 669e1cfd407e4d87a63e6832d9c6fa1f
0112 20:51:50.946328 (+ 4517us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.959457 (+ 13129us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.006274 (+ 46817us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.026891 (+ 20617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588626}
I20250112 20:51:51.029244 1185 tablet_replica.cc:331] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.031424 1185 raft_consensus.cc:2238] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.032754 1185 raft_consensus.cc:2267] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:51.033010 983 consensus_peers.cc:487] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f875f075e98e45499291b519caf03184. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.037060 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1050) took 1673 ms. Trace:
I20250112 20:51:51.037149 1186 rpcz_store.cc:276] 0112 20:51:49.363240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.363401 (+ 161us) service_pool.cc:225] Handling call
0112 20:51:51.017614 (+1654213us) ts_tablet_manager.cc:1145] Deleting tablet 9570269368e14b1abeaffdcc741c6f3a
0112 20:51:51.017623 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.023473 (+ 5850us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.026163 (+ 2690us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.037040 (+ 10877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654009}
I20250112 20:51:51.037436 1186 tablet_replica.cc:331] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.037503 1185 ts_tablet_manager.cc:1905] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.039404 3287 tablet.cc:1721] T a218ef5825424b1b84c55b38b9b9023d P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:51.042083 1186 raft_consensus.cc:2238] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:51.044787 847 leader_election.cc:343] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.045873 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80791d8582446b89adf212d3d6507eb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:51.046139 758 raft_consensus.cc:3054] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.046485 1186 pending_rounds.cc:62] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:51.046567 1186 pending_rounds.cc:69] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585088551563264 op_type: NO_OP noop_request { }
I20250112 20:51:51.046700 1186 raft_consensus.cc:2883] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:51.046808 1186 raft_consensus.cc:2267] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.047569 1186 ts_tablet_manager.cc:1905] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.049206 758 raft_consensus.cc:2463] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.051566 1185 ts_tablet_manager.cc:1918] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.051694 1185 log.cc:1198] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/924082716c1e4d9ea6fbe97ae93232bd
I20250112 20:51:51.052050 1185 ts_tablet_manager.cc:1939] T 924082716c1e4d9ea6fbe97ae93232bd P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:51.052343 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1019) took 2703 ms. Trace:
I20250112 20:51:51.059645 1178 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.059944 1178 ts_tablet_manager.cc:1397] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.004s
I20250112 20:51:51.061635 1178 raft_consensus.cc:357] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.062511 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8074b00eda8d434faaff4b98ae923ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.062331 1178 raft_consensus.cc:383] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.063221 1178 raft_consensus.cc:738] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:51.064162 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30321942214949b3bcb727d49f4a34d6 consensus state has no leader
I20250112 20:51:51.068984 2447 ts_tablet_manager.cc:1918] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.073988 1186 ts_tablet_manager.cc:1918] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:51.076246 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9854d410371b442c81705698f427d8b0 consensus state has no leader
I20250112 20:51:51.065078 1178 consensus_queue.cc:260] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.077765 1178 ts_tablet_manager.cc:1428] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250112 20:51:51.078267 1178 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.081432 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2734d2857154455f9cbdcd8e5f06b1f2" } new_table_name: "default.Table_436e05c77330405c813045f5df876b88" modify_external_catalogs: true
I20250112 20:51:51.085429 847 leader_election.cc:304] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.085850 3178 raft_consensus.cc:2798] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.085983 3178 raft_consensus.cc:695] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:51.086247 3178 consensus_queue.cc:237] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.008304 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 49b8cc99742f4f9ba68eb1aa17cab248 (table default.Table_25113f809e324283a5b53261123398f1 [id=e3b1932f0a7c43fbb5a8ae879c51b0e6]) successfully deleted
W20250112 20:51:51.086966 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 78675.4us
I20250112 20:51:51.089155 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 669e1cfd407e4d87a63e6832d9c6fa1f (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
I20250112 20:51:51.053575 3287 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:51.095803 1178 tablet_bootstrap.cc:654] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.069265 1164 tablet_bootstrap.cc:700] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Time spent opening tablet: real 0.176s user 0.001s sys 0.000s
I20250112 20:51:51.082489 3321 tablet.cc:1721] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 20:51:51.088886 710 leader_election.cc:343] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.099406 1164 tablet_bootstrap.cc:654] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.100095 3268 consensus_queue.cc:1035] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.100325 1185 rpcz_store.cc:276] 0112 20:51:48.349155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.355284 (+ 6129us) service_pool.cc:225] Handling call
0112 20:51:51.029199 (+2673915us) ts_tablet_manager.cc:1145] Deleting tablet 924082716c1e4d9ea6fbe97ae93232bd
0112 20:51:51.029210 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.048735 (+ 19525us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.051486 (+ 2751us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.052329 (+ 843us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3840,"tablet-delete.queue_time_us":2673752}
I20250112 20:51:51.100652 1185 tablet_replica.cc:331] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.100857 1185 raft_consensus.cc:2238] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.101014 1185 raft_consensus.cc:2267] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.101486 1185 ts_tablet_manager.cc:1905] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:51.104096 710 consensus_peers.cc:487] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b74aaf7f5b4c87acbefaa4df5c10d2. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.107820 710 leader_election.cc:304] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.108134 3114 raft_consensus.cc:2798] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.108254 3114 raft_consensus.cc:695] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:51.108529 3114 consensus_queue.cc:237] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.091732 3325 tablet.cc:1721] T 97d32367b39a4cce8371377ec7449586 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.114022 2447 log.cc:1198] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/669e1cfd407e4d87a63e6832d9c6fa1f
I20250112 20:51:51.114760 3097 tablet.cc:1721] T c11a892641554e4db9bdd558068cac40 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.115561 3250 tablet.cc:1721] T c11a892641554e4db9bdd558068cac40 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.116981 1163 tablet_bootstrap.cc:492] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.117151 1163 ts_tablet_manager.cc:1397] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.267s user 0.004s sys 0.002s
I20250112 20:51:51.117780 1163 raft_consensus.cc:357] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.117982 1163 raft_consensus.cc:383] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.118041 1163 raft_consensus.cc:738] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.118386 1163 consensus_queue.cc:260] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.119077 1163 ts_tablet_manager.cc:1428] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:51.125797 1186 log.cc:1198] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/e8bea4d83ea644a0930a29f61e7a5fc8
I20250112 20:51:51.126188 1186 ts_tablet_manager.cc:1939] T e8bea4d83ea644a0930a29f61e7a5fc8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.126889 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1044) took 1792 ms. Trace:
I20250112 20:51:51.126997 1186 rpcz_store.cc:276] 0112 20:51:49.334602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.334657 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:51.037395 (+1702738us) ts_tablet_manager.cc:1145] Deleting tablet e8bea4d83ea644a0930a29f61e7a5fc8
0112 20:51:51.037405 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.042412 (+ 5007us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x616000280f54. stack: 00007fd9ea48afd6 00007fd9ea48ad90 00007fd9e946a48d 000000000096d544 000000000096d50d 000000000096ceda 00007fd9f5d2ded0 00007fd9f7022463 00007fd9f9ac6eb7 00007fd9f9af9584 00007fd9f9ada690 00007fd9f9ada41d 0000000000a71ecc 00007fd9ea4d1b64 00007fd9ea4d518b 00007fd9ea4d5008
0112 20:51:51.057440 (+ 15028us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.073880 (+ 16440us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.126873 (+ 52993us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3926,"spinlock_wait_cycles":10194176,"tablet-delete.queue_time_us":1660394}
I20250112 20:51:51.127410 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1043) took 1792 ms. Trace:
I20250112 20:51:51.127476 1186 rpcz_store.cc:276] 0112 20:51:49.334509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.335041 (+ 532us) service_pool.cc:225] Handling call
0112 20:51:51.127286 (+1792245us) ts_tablet_manager.cc:1145] Deleting tablet 532ccaeaf38c41a6ab26f65411d31418
0112 20:51:51.127298 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.127397 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742250}
W20250112 20:51:51.127671 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c17e8808d62d4b7da88bffe7d7569568 (table default.Table_ebc936227805419b91c80e6d8a21e144 [id=b5c55f1217cb4bada8cba93ada44f7f4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c17e8808d62d4b7da88bffe7d7569568 consensus state has no leader
I20250112 20:51:51.127717 1186 tablet_replica.cc:331] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.127892 1186 raft_consensus.cc:2238] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.128063 1186 raft_consensus.cc:2267] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.128181 2447 ts_tablet_manager.cc:1939] T 669e1cfd407e4d87a63e6832d9c6fa1f P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.128171 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 226ea7f38f1445c598676715a3036227 (table default.Table_1344f1b683b941899cf95663d76ffd18 [id=2e501409d15843689ad2eaa69f9e8392]) successfully deleted
W20250112 20:51:51.128407 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6b440da020d947dda2553d65dd6a4bdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b440da020d947dda2553d65dd6a4bdf already in progress: creating tablet
W20250112 20:51:51.128681 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 146994989d6d4ce8a12e1b8677d7feb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 146994989d6d4ce8a12e1b8677d7feb7 already in progress: creating tablet
I20250112 20:51:51.128940 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 71743c73b6f24da9b14a3d4148117813 (table default.Table_d8d06d1114e0487d8f3097053759ceb8 [id=a463efa09ff246029307751a49e80170]) successfully deleted
W20250112 20:51:51.129212 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 2c8c7784e5534c2c91f124dffd24d5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8c7784e5534c2c91f124dffd24d5c5 already in progress: creating tablet
W20250112 20:51:51.129585 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cfb833659c23431daaab008dcdf1de43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb833659c23431daaab008dcdf1de43 already in progress: creating tablet
I20250112 20:51:51.129812 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b85b9e82f9e847c4acfeecb82201e694 (table default.Table_7927297ff7264be5a6391d2a58f38280 [id=8530d1e179394e18ad4464ae2187dcf9]) successfully deleted
I20250112 20:51:51.130045 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e0fd4177f9394a5a82d4f5e02369b3b6 (table default.Table_cba467c888b84da6964f049247721310 [id=c0f4f9a776794374b5cfc063becc2876]) successfully deleted
W20250112 20:51:51.130326 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet db9bab2477a94c169817d9de8aaad1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9bab2477a94c169817d9de8aaad1c0 already in progress: creating tablet
W20250112 20:51:51.130559 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3ea9425f3e7b46288c81dc8cdade99b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ea9425f3e7b46288c81dc8cdade99b2 already in progress: creating tablet
I20250112 20:51:51.130800 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (table default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]) successfully deleted
I20250112 20:51:51.131268 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 45484f8e48524b86a6f418b88a71ac18 (table default.Table_fbe3a476695f40ec8737a0e3e4ad731f [id=eba58b5d38e8443e842cca705dc51aeb]) successfully deleted
I20250112 20:51:51.136376 1185 ts_tablet_manager.cc:1918] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.138331 1186 ts_tablet_manager.cc:1905] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.144073 1185 log.cc:1198] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/dc3a9740415140fe8864d54f477af954
I20250112 20:51:51.144639 1185 ts_tablet_manager.cc:1939] T dc3a9740415140fe8864d54f477af954 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:51.146502 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1018) took 2797 ms. Trace:
I20250112 20:51:51.151352 1186 ts_tablet_manager.cc:1918] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.151463 1186 log.cc:1198] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5f264d84ae4846d1afaaabc41d5789f4
I20250112 20:51:51.152493 3121 consensus_queue.cc:1035] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.151780 1186 ts_tablet_manager.cc:1939] T 5f264d84ae4846d1afaaabc41d5789f4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.165591 3121 consensus_queue.cc:1035] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.172425 1163 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:51.172685 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 970) took 2771 ms. Trace:
I20250112 20:51:51.157186 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1045) took 1822 ms. Trace:
I20250112 20:51:51.172834 1186 rpcz_store.cc:276] 0112 20:51:49.334846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.335116 (+ 270us) service_pool.cc:225] Handling call
0112 20:51:51.127680 (+1792564us) ts_tablet_manager.cc:1145] Deleting tablet 5f264d84ae4846d1afaaabc41d5789f4
0112 20:51:51.127688 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.146485 (+ 18797us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.151273 (+ 4788us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.157165 (+ 5892us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742509}
I20250112 20:51:51.172914 2447 rpcz_store.cc:276] 0112 20:51:48.401402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.492969 (+ 91567us) service_pool.cc:225] Handling call
0112 20:51:50.803807 (+2310838us) ts_tablet_manager.cc:1145] Deleting tablet 669e1cfd407e4d87a63e6832d9c6fa1f
0112 20:51:50.803818 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:50.857944 (+ 54126us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.068881 (+210937us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.172631 (+103750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310636}
I20250112 20:51:51.173158 1186 tablet_replica.cc:331] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.173197 2447 tablet_replica.cc:331] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.173345 1186 raft_consensus.cc:2238] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.173377 2447 raft_consensus.cc:2238] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.173506 1186 raft_consensus.cc:2267] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.173534 2447 raft_consensus.cc:2267] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.174649 1186 ts_tablet_manager.cc:1905] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.175191 2447 ts_tablet_manager.cc:1905] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.175254 1163 tablet_bootstrap.cc:654] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.179805 1185 rpcz_store.cc:276] 0112 20:51:48.349014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.360617 (+ 11603us) service_pool.cc:225] Handling call
0112 20:51:51.100609 (+2739992us) ts_tablet_manager.cc:1145] Deleting tablet dc3a9740415140fe8864d54f477af954
0112 20:51:51.100618 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.112466 (+ 11848us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.136279 (+ 23813us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.146483 (+ 10204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2739786}
I20250112 20:51:51.180192 1185 tablet_replica.cc:331] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.181618 2447 ts_tablet_manager.cc:1918] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.181815 2447 log.cc:1198] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/924082716c1e4d9ea6fbe97ae93232bd
I20250112 20:51:51.182310 2447 ts_tablet_manager.cc:1939] T 924082716c1e4d9ea6fbe97ae93232bd P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.183096 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 971) took 2690 ms. Trace:
I20250112 20:51:51.183235 2447 rpcz_store.cc:276] 0112 20:51:48.493098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493145 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:51.173141 (+2679996us) ts_tablet_manager.cc:1145] Deleting tablet 924082716c1e4d9ea6fbe97ae93232bd
0112 20:51:51.173149 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.178151 (+ 5002us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.181520 (+ 3369us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.183059 (+ 1539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2679811}
I20250112 20:51:51.183746 2447 tablet_replica.cc:331] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.184012 2447 raft_consensus.cc:2238] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.184384 2447 raft_consensus.cc:2267] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.184976 2447 ts_tablet_manager.cc:1905] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.190218 1164 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.190438 1164 ts_tablet_manager.cc:1397] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.298s user 0.009s sys 0.006s
I20250112 20:51:51.191059 1164 raft_consensus.cc:357] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.191282 1164 raft_consensus.cc:383] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.191378 1164 raft_consensus.cc:738] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.191634 1164 consensus_queue.cc:260] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.192334 1164 ts_tablet_manager.cc:1428] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:51.192742 1164 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.194435 1164 tablet_bootstrap.cc:654] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.195293 1186 ts_tablet_manager.cc:1918] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.195410 1186 log.cc:1198] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/104d6b4effb84314a6b2b9db883b4366
I20250112 20:51:51.178272 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d571de1e2b104ba480cdaf2523b54599
I20250112 20:51:51.179812 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_84f9735e457340a490e02f9c6983d2e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.199522 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84f9735e457340a490e02f9c6983d2e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:51.201138 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 69846.9us
W20250112 20:51:51.201479 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet dfc63eff44414906a027a8208b5cc5a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc63eff44414906a027a8208b5cc5a9 already in progress: creating tablet
W20250112 20:51:51.201805 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ed925a5ef3a54f819ca292a4e7967b33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed925a5ef3a54f819ca292a4e7967b33 already in progress: creating tablet
W20250112 20:51:51.202178 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet cbc22207e9204b73877c74f61ebba8a8 because the tablet was not found. No further retry: Not found: Tablet not found: cbc22207e9204b73877c74f61ebba8a8
W20250112 20:51:51.202454 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3846863c74f64923bda6e5a19e7f2770 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3846863c74f64923bda6e5a19e7f2770 already in progress: creating tablet
I20250112 20:51:51.203023 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9880e7d303b54d959bca4e8464a6b30c (table default.Table_0cf0e2c39db54427949f7adfbec95e0d [id=537bf8b60c0249efb50df78260ed60c3]) successfully deleted
I20250112 20:51:51.204365 1186 ts_tablet_manager.cc:1939] T 104d6b4effb84314a6b2b9db883b4366 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.206898 3265 consensus_queue.cc:1035] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.213241 1164 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.213819 1164 ts_tablet_manager.cc:1397] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250112 20:51:51.215730 1164 raft_consensus.cc:357] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.216141 1164 raft_consensus.cc:383] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.216286 1164 raft_consensus.cc:738] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.216565 1164 consensus_queue.cc:260] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.217530 1164 ts_tablet_manager.cc:1428] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:51.218137 1164 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.219786 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1046) took 1884 ms. Trace:
I20250112 20:51:51.219887 1186 rpcz_store.cc:276] 0112 20:51:49.334941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.335165 (+ 224us) service_pool.cc:225] Handling call
0112 20:51:51.173110 (+1837945us) ts_tablet_manager.cc:1145] Deleting tablet 104d6b4effb84314a6b2b9db883b4366
0112 20:51:51.173119 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.177669 (+ 4550us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.195170 (+ 17501us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.219765 (+ 24595us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787840}
I20250112 20:51:51.220326 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1042) took 1885 ms. Trace:
I20250112 20:51:51.220430 1186 rpcz_store.cc:276] 0112 20:51:49.334414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.334988 (+ 574us) service_pool.cc:225] Handling call
0112 20:51:51.220193 (+1885205us) ts_tablet_manager.cc:1145] Deleting tablet 0866871b9e1542e8ab30505c68f7dea9
0112 20:51:51.220211 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.220304 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834864}
I20250112 20:51:51.220767 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1056) took 1815 ms. Trace:
I20250112 20:51:51.220875 1186 rpcz_store.cc:276] 0112 20:51:49.405203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.408858 (+ 3655us) service_pool.cc:225] Handling call
0112 20:51:51.220675 (+1811817us) ts_tablet_manager.cc:1145] Deleting tablet 7959f3ed319f4078ab747d490dce7fe1
0112 20:51:51.220685 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.220749 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1811637}
I20250112 20:51:51.221154 1186 tablet_replica.cc:331] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.221388 1186 raft_consensus.cc:2238] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.221531 3325 tablet.cc:1721] T c11a892641554e4db9bdd558068cac40 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.221648 1186 pending_rounds.cc:62] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:51.221691 1186 pending_rounds.cc:69] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585095011545088 op_type: NO_OP noop_request { }
I20250112 20:51:51.221796 1186 raft_consensus.cc:2883] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:51.221884 1186 raft_consensus.cc:2267] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.223186 1164 tablet_bootstrap.cc:654] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.229013 1186 ts_tablet_manager.cc:1905] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.237443 1178 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.238778 1178 ts_tablet_manager.cc:1397] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.007s
I20250112 20:51:51.241974 1185 raft_consensus.cc:2238] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.242403 1185 raft_consensus.cc:2267] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.243257 1185 ts_tablet_manager.cc:1905] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.243978 2447 ts_tablet_manager.cc:1918] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.244065 2447 log.cc:1198] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/dc3a9740415140fe8864d54f477af954
I20250112 20:51:51.244427 2447 ts_tablet_manager.cc:1939] T dc3a9740415140fe8864d54f477af954 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.244700 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 972) took 2751 ms. Trace:
I20250112 20:51:51.244781 2447 rpcz_store.cc:276] 0112 20:51:48.493219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493257 (+ 38us) service_pool.cc:225] Handling call
0112 20:51:51.183688 (+2690431us) ts_tablet_manager.cc:1145] Deleting tablet dc3a9740415140fe8864d54f477af954
0112 20:51:51.183698 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.239502 (+ 55804us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.243871 (+ 4369us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.244685 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2690329}
I20250112 20:51:51.245101 2447 tablet_replica.cc:331] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.245258 2447 raft_consensus.cc:2238] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.245537 2447 raft_consensus.cc:2267] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.246033 2447 ts_tablet_manager.cc:1905] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.249576 1185 ts_tablet_manager.cc:1918] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.249748 1185 log.cc:1198] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/7f807071e53a4f0e8629b1dfa6d26b9d
I20250112 20:51:51.250177 1185 ts_tablet_manager.cc:1939] T 7f807071e53a4f0e8629b1dfa6d26b9d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:51.252255 2447 ts_tablet_manager.cc:1918] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.252416 2447 log.cc:1198] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/3dba3d9315db4b9292288ee62c2244bc
I20250112 20:51:51.252830 2447 ts_tablet_manager.cc:1939] T 3dba3d9315db4b9292288ee62c2244bc P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.253144 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 973) took 2759 ms. Trace:
I20250112 20:51:51.253257 2447 rpcz_store.cc:276] 0112 20:51:48.493324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493374 (+ 50us) service_pool.cc:225] Handling call
0112 20:51:51.245061 (+2751687us) ts_tablet_manager.cc:1145] Deleting tablet 3dba3d9315db4b9292288ee62c2244bc
0112 20:51:51.245069 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.249273 (+ 4204us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.252184 (+ 2911us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.253131 (+ 947us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2751596}
I20250112 20:51:51.254333 2447 tablet_replica.cc:331] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.254443 1186 ts_tablet_manager.cc:1918] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.254465 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1017) took 2905 ms. Trace:
I20250112 20:51:51.254546 1186 log.cc:1198] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d80791d8582446b89adf212d3d6507eb
I20250112 20:51:51.254627 2447 raft_consensus.cc:2238] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.254845 2447 raft_consensus.cc:2267] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.254879 1186 ts_tablet_manager.cc:1939] T d80791d8582446b89adf212d3d6507eb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.254662 1185 rpcz_store.cc:276] 0112 20:51:48.348869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.360857 (+ 11988us) service_pool.cc:225] Handling call
0112 20:51:51.180140 (+2819283us) ts_tablet_manager.cc:1145] Deleting tablet 7f807071e53a4f0e8629b1dfa6d26b9d
0112 20:51:51.180152 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.242306 (+ 62154us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x6160000a6054. stack: 00007f1a9e448fd6 00007f1a9e448d90 00007f1a9d42848d 000000000096d544 000000000096d50d 000000000096ceda 00007f1aa9cebed0 00007f1aaafe0463 00007f1aada84eb7 00007f1aadab7584 00007f1aada98690 00007f1aada9841d 0000000000a71ecc 00007f1a9e48fb64 00007f1a9e49318b 00007f1a9e493008
0112 20:51:51.246731 (+ 4425us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.249501 (+ 2770us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.254447 (+ 4946us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":141623168,"tablet-delete.queue_time_us":2819053}
I20250112 20:51:51.255373 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1024) took 2840 ms. Trace:
I20250112 20:51:51.255507 2447 ts_tablet_manager.cc:1905] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.255533 1185 rpcz_store.cc:276] 0112 20:51:48.415342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.415484 (+ 142us) service_pool.cc:225] Handling call
0112 20:51:51.255234 (+2839750us) ts_tablet_manager.cc:1145] Deleting tablet 3ea9425f3e7b46288c81dc8cdade99b2
0112 20:51:51.255246 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.255357 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2829884}
I20250112 20:51:51.255824 1164 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.255967 1164 ts_tablet_manager.cc:1397] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20250112 20:51:51.256286 1185 tablet_replica.cc:331] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.256573 1185 raft_consensus.cc:2238] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.256639 1164 raft_consensus.cc:357] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.256848 1164 raft_consensus.cc:383] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.256829 1185 raft_consensus.cc:2267] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.257542 1185 ts_tablet_manager.cc:1905] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.260604 3121 raft_consensus.cc:491] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.251792 446 master_service.cc:946] ReplaceTablet: received request to replace tablet b496462873e44ceab2ec3c95cf63fc5a from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:51.260771 3121 raft_consensus.cc:513] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:51.252140 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c950294baf3046a09d2202912ab8c8c8 consensus state has no leader
I20250112 20:51:51.261121 2447 ts_tablet_manager.cc:1918] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:51.261291 2447 log.cc:1198] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/c8d24bccf682483aa9b273fc1ce847da
W20250112 20:51:51.261322 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 58276us
I20250112 20:51:51.261597 3121 leader_election.cc:290] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:51.261693 2447 ts_tablet_manager.cc:1939] T c8d24bccf682483aa9b273fc1ce847da P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.256903 1164 raft_consensus.cc:738] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.262084 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 9570269368e14b1abeaffdcc741c6f3a (table default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]) successfully deleted
I20250112 20:51:51.262370 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet e8bea4d83ea644a0930a29f61e7a5fc8 (table default.Table_92f355d4058f40b6b0979ff3f3f1af14 [id=2049ad7ffcee4affa56490ef29e949da]) successfully deleted
I20250112 20:51:51.262296 1164 consensus_queue.cc:260] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.262468 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:51.262562 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 532ccaeaf38c41a6ab26f65411d31418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ccaeaf38c41a6ab26f65411d31418 already in progress: creating tablet
I20250112 20:51:51.262741 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5f264d84ae4846d1afaaabc41d5789f4 (table default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]) successfully deleted
I20250112 20:51:51.262820 1164 ts_tablet_manager.cc:1428] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 20:51:51.262907 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 104d6b4effb84314a6b2b9db883b4366 (table default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]) successfully deleted
I20250112 20:51:51.263077 1164 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.263142 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:51.263691 710 leader_election.cc:343] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:51.263945 710 leader_election.cc:343] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.264117 710 leader_election.cc:304] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:51.264190 1163 tablet_bootstrap.cc:492] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.264449 3265 raft_consensus.cc:2743] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.264663 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 976) took 2771 ms. Trace:
I20250112 20:51:51.270851 2447 rpcz_store.cc:276] 0112 20:51:48.493576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493611 (+ 35us) service_pool.cc:225] Handling call
0112 20:51:51.254289 (+2760678us) ts_tablet_manager.cc:1145] Deleting tablet c8d24bccf682483aa9b273fc1ce847da
0112 20:51:51.254299 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.258371 (+ 4072us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.261053 (+ 2682us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.264646 (+ 3593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2760526}
I20250112 20:51:51.271245 2447 tablet_replica.cc:331] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.271440 2447 raft_consensus.cc:2238] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.271765 2447 raft_consensus.cc:2267] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.272599 2447 ts_tablet_manager.cc:1905] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.274461 1164 tablet_bootstrap.cc:654] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.275079 1178 raft_consensus.cc:357] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.277501 1178 raft_consensus.cc:383] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.278229 1178 raft_consensus.cc:738] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.278898 2447 ts_tablet_manager.cc:1918] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.279004 2447 log.cc:1198] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1fd34e829d0645ea80daaff5d27cb266
I20250112 20:51:51.279340 2447 ts_tablet_manager.cc:1939] T 1fd34e829d0645ea80daaff5d27cb266 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.279628 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 977) took 2785 ms. Trace:
I20250112 20:51:51.280405 1163 ts_tablet_manager.cc:1397] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20250112 20:51:51.278993 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1083) took 1426 ms. Trace:
I20250112 20:51:51.281095 1186 rpcz_store.cc:276] 0112 20:51:49.852287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.863466 (+ 11179us) service_pool.cc:225] Handling call
0112 20:51:51.221111 (+1357645us) ts_tablet_manager.cc:1145] Deleting tablet d80791d8582446b89adf212d3d6507eb
0112 20:51:51.221121 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.235931 (+ 14810us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.254354 (+ 18423us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.278974 (+ 24620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357479}
I20250112 20:51:51.281473 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1084) took 1429 ms. Trace:
I20250112 20:51:51.281545 1186 rpcz_store.cc:276] 0112 20:51:49.852377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.863640 (+ 11263us) service_pool.cc:225] Handling call
0112 20:51:51.281365 (+1417725us) ts_tablet_manager.cc:1145] Deleting tablet ced81345e31a4634bb21e3702ad69c93
0112 20:51:51.281374 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.281460 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417597}
I20250112 20:51:51.281831 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1086) took 1281 ms. Trace:
I20250112 20:51:51.281919 1186 rpcz_store.cc:276] 0112 20:51:50.000684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.000844 (+ 160us) service_pool.cc:225] Handling call
0112 20:51:51.281743 (+1280899us) ts_tablet_manager.cc:1145] Deleting tablet c3812f8aa9ba422f85f331648a1ec02e
0112 20:51:51.281751 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.281818 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280691}
I20250112 20:51:51.282192 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1085) took 1282 ms. Trace:
I20250112 20:51:51.282279 1186 rpcz_store.cc:276] 0112 20:51:50.000133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.001109 (+ 976us) service_pool.cc:225] Handling call
0112 20:51:51.282108 (+1280999us) ts_tablet_manager.cc:1145] Deleting tablet 7623bebb539e485aa508e95a35225979
0112 20:51:51.282116 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.282182 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280818}
I20250112 20:51:51.282524 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1089) took 1281 ms. Trace:
I20250112 20:51:51.282589 1186 rpcz_store.cc:276] 0112 20:51:50.001408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.001669 (+ 261us) service_pool.cc:225] Handling call
0112 20:51:51.282450 (+1280781us) ts_tablet_manager.cc:1145] Deleting tablet c90c8dd4dfb14507accb82828be512f0
0112 20:51:51.282457 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.282513 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280562}
I20250112 20:51:51.282845 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1090) took 1281 ms. Trace:
I20250112 20:51:51.282907 1186 rpcz_store.cc:276] 0112 20:51:50.001518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.001801 (+ 283us) service_pool.cc:225] Handling call
0112 20:51:51.282774 (+1280973us) ts_tablet_manager.cc:1145] Deleting tablet 961a72a9d8a643cea743f3f021bd5a03
0112 20:51:51.282782 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.282835 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280784}
I20250112 20:51:51.283183 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1092) took 1281 ms. Trace:
I20250112 20:51:51.283257 1186 rpcz_store.cc:276] 0112 20:51:50.001760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.002201 (+ 441us) service_pool.cc:225] Handling call
0112 20:51:51.283113 (+1280912us) ts_tablet_manager.cc:1145] Deleting tablet a5819f1c6329433183492855c3b08476
0112 20:51:51.283120 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.283173 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280739}
I20250112 20:51:51.283458 1186 tablet_replica.cc:331] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.283624 1186 raft_consensus.cc:2238] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.280766 1178 consensus_queue.cc:260] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.285212 1163 raft_consensus.cc:357] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.285396 1163 raft_consensus.cc:383] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.285461 1163 raft_consensus.cc:738] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.285642 1163 consensus_queue.cc:260] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.286201 1163 ts_tablet_manager.cc:1428] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20250112 20:51:51.286450 1163 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:51.287678 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 0866871b9e1542e8ab30505c68f7dea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0866871b9e1542e8ab30505c68f7dea9 already in progress: creating tablet
W20250112 20:51:51.287983 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 7959f3ed319f4078ab747d490dce7fe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7959f3ed319f4078ab747d490dce7fe1 already in progress: creating tablet
I20250112 20:51:51.288198 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d80791d8582446b89adf212d3d6507eb (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
W20250112 20:51:51.288424 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet ced81345e31a4634bb21e3702ad69c93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced81345e31a4634bb21e3702ad69c93 already in progress: creating tablet
W20250112 20:51:51.288650 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c3812f8aa9ba422f85f331648a1ec02e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3812f8aa9ba422f85f331648a1ec02e already in progress: creating tablet
W20250112 20:51:51.288924 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 7623bebb539e485aa508e95a35225979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7623bebb539e485aa508e95a35225979 already in progress: creating tablet
I20250112 20:51:51.283890 1186 raft_consensus.cc:2267] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.290702 1178 ts_tablet_manager.cc:1428] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20250112 20:51:51.290856 1186 ts_tablet_manager.cc:1905] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.279706 2447 rpcz_store.cc:276] 0112 20:51:48.493654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493736 (+ 82us) service_pool.cc:225] Handling call
0112 20:51:51.271192 (+2777456us) ts_tablet_manager.cc:1145] Deleting tablet 1fd34e829d0645ea80daaff5d27cb266
0112 20:51:51.271202 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.275987 (+ 4785us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.278809 (+ 2822us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.279614 (+ 805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2777366}
I20250112 20:51:51.295578 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 978) took 2801 ms. Trace:
I20250112 20:51:51.295655 2447 rpcz_store.cc:276] 0112 20:51:48.493734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493838 (+ 104us) service_pool.cc:225] Handling call
0112 20:51:51.295459 (+2801621us) ts_tablet_manager.cc:1145] Deleting tablet db9bab2477a94c169817d9de8aaad1c0
0112 20:51:51.295470 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.295561 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2801505}
I20250112 20:51:51.295976 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 979) took 2802 ms. Trace:
I20250112 20:51:51.296041 2447 rpcz_store.cc:276] 0112 20:51:48.493846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493992 (+ 146us) service_pool.cc:225] Handling call
0112 20:51:51.295886 (+2801894us) ts_tablet_manager.cc:1145] Deleting tablet 3ea9425f3e7b46288c81dc8cdade99b2
0112 20:51:51.295895 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.295963 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2801800}
I20250112 20:51:51.296265 2447 tablet_replica.cc:331] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.296430 2447 raft_consensus.cc:2238] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.296557 2447 raft_consensus.cc:2267] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.297071 2447 ts_tablet_manager.cc:1905] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.297693 1185 ts_tablet_manager.cc:1918] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:51.297909 1185 log.cc:1198] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/45484f8e48524b86a6f418b88a71ac18
I20250112 20:51:51.298532 1185 ts_tablet_manager.cc:1939] T 45484f8e48524b86a6f418b88a71ac18 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:51.299875 1178 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.301402 1178 tablet_bootstrap.cc:654] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.312824 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:51.314492 1186 ts_tablet_manager.cc:1918] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:51.314656 1186 log.cc:1198] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/53551931b23d43da89e9193cdf034a63
I20250112 20:51:51.315023 1186 ts_tablet_manager.cc:1939] T 53551931b23d43da89e9193cdf034a63 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.315325 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1093) took 1313 ms. Trace:
I20250112 20:51:51.315415 1186 rpcz_store.cc:276] 0112 20:51:50.001895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.002600 (+ 705us) service_pool.cc:225] Handling call
0112 20:51:51.283426 (+1280826us) ts_tablet_manager.cc:1145] Deleting tablet 53551931b23d43da89e9193cdf034a63
0112 20:51:51.283433 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.293987 (+ 10554us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.314399 (+ 20412us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.315314 (+ 915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280661}
I20250112 20:51:51.315850 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1100) took 1149 ms. Trace:
I20250112 20:51:51.318331 1186 rpcz_store.cc:276] 0112 20:51:50.166290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.168455 (+ 2165us) service_pool.cc:225] Handling call
0112 20:51:51.315741 (+1147286us) ts_tablet_manager.cc:1145] Deleting tablet 8bc545822e98460e815df4e6cad455b4
0112 20:51:51.315750 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.315837 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147104}
I20250112 20:51:51.318763 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b95e22f892d24fe390619250ef5fba8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.319315 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1105) took 1092 ms. Trace:
I20250112 20:51:51.319456 1186 rpcz_store.cc:276] 0112 20:51:50.226778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.228740 (+ 1962us) service_pool.cc:225] Handling call
0112 20:51:51.319202 (+1090462us) ts_tablet_manager.cc:1145] Deleting tablet 5065a94bb13c419e9d9b8eaca72f5052
0112 20:51:51.319220 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.319303 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090249}
I20250112 20:51:51.319836 1164 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.320452 1164 ts_tablet_manager.cc:1397] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.000s
I20250112 20:51:51.321532 1164 raft_consensus.cc:357] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.321772 1164 raft_consensus.cc:383] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.321873 1164 raft_consensus.cc:738] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.322141 1164 consensus_queue.cc:260] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.323076 1164 ts_tablet_manager.cc:1428] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:51.323573 1164 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.325328 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1106) took 1019 ms. Trace:
I20250112 20:51:51.325402 1186 rpcz_store.cc:276] 0112 20:51:50.305600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.307178 (+ 1578us) service_pool.cc:225] Handling call
0112 20:51:51.325193 (+1018015us) ts_tablet_manager.cc:1145] Deleting tablet 6181b166b5cb429da3ebba23cf267942
0112 20:51:51.325202 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.325315 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017832}
I20250112 20:51:51.325632 1186 tablet_replica.cc:331] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.325801 1186 raft_consensus.cc:2238] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.325976 1186 raft_consensus.cc:2267] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.330307 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1025) took 2891 ms. Trace:
I20250112 20:51:51.330571 1186 ts_tablet_manager.cc:1905] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.330492 1185 rpcz_store.cc:276] 0112 20:51:48.438510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.443257 (+ 4747us) service_pool.cc:225] Handling call
0112 20:51:51.256150 (+2812893us) ts_tablet_manager.cc:1145] Deleting tablet 45484f8e48524b86a6f418b88a71ac18
0112 20:51:51.256162 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.260402 (+ 4240us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.297556 (+ 37154us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.330287 (+ 32731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2812671}
I20250112 20:51:51.331507 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1027) took 2667 ms. Trace:
I20250112 20:51:51.331708 1185 rpcz_store.cc:276] 0112 20:51:48.664047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.664101 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:51.331315 (+2667214us) ts_tablet_manager.cc:1145] Deleting tablet dfc63eff44414906a027a8208b5cc5a9
0112 20:51:51.331325 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.331489 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2666948}
I20250112 20:51:51.332458 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1028) took 2582 ms. Trace:
I20250112 20:51:51.332703 1185 rpcz_store.cc:276] 0112 20:51:48.750261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.750331 (+ 70us) service_pool.cc:225] Handling call
0112 20:51:51.332266 (+2581935us) ts_tablet_manager.cc:1145] Deleting tablet ed925a5ef3a54f819ca292a4e7967b33
0112 20:51:51.332275 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.332437 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2581727}
I20250112 20:51:51.333606 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1029) took 2500 ms. Trace:
I20250112 20:51:51.333827 1185 rpcz_store.cc:276] 0112 20:51:48.833467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.833545 (+ 78us) service_pool.cc:225] Handling call
0112 20:51:51.333390 (+2499845us) ts_tablet_manager.cc:1145] Deleting tablet 3846863c74f64923bda6e5a19e7f2770
0112 20:51:51.333404 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.333561 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2445837}
I20250112 20:51:51.334623 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 1e584b761d234c5aa0ab875acd8e1e05 (table default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a]). Expected version 4 got 0
I20250112 20:51:51.334640 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1044) took 2255 ms. Trace:
W20250112 20:51:51.335994 983 consensus_peers.cc:487] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ca8fc4a45fe448b919ac001a7980deb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.336365 1186 ts_tablet_manager.cc:1918] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:51.336465 1186 log.cc:1198] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/db9178cf782d4bd28eba90cc03796885
I20250112 20:51:51.337394 1178 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.337554 1178 ts_tablet_manager.cc:1397] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250112 20:51:51.338227 1178 raft_consensus.cc:357] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.338474 1178 raft_consensus.cc:383] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.338640 1178 raft_consensus.cc:738] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.339076 1178 consensus_queue.cc:260] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.340240 1178 ts_tablet_manager.cc:1428] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:51.344161 3258 log.cc:826] T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:51.345556 1186 ts_tablet_manager.cc:1939] T db9178cf782d4bd28eba90cc03796885 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:51.352888 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet c90c8dd4dfb14507accb82828be512f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c90c8dd4dfb14507accb82828be512f0 already in progress: creating tablet
W20250112 20:51:51.353240 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 64110.4us
W20250112 20:51:51.353614 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 961a72a9d8a643cea743f3f021bd5a03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961a72a9d8a643cea743f3f021bd5a03 already in progress: creating tablet
W20250112 20:51:51.353956 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a5819f1c6329433183492855c3b08476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5819f1c6329433183492855c3b08476 already in progress: creating tablet
I20250112 20:51:51.354216 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]) successfully deleted
W20250112 20:51:51.354465 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8bc545822e98460e815df4e6cad455b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc545822e98460e815df4e6cad455b4 already in progress: creating tablet
W20250112 20:51:51.354816 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 5065a94bb13c419e9d9b8eaca72f5052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5065a94bb13c419e9d9b8eaca72f5052 already in progress: creating tablet
W20250112 20:51:51.355098 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 6181b166b5cb429da3ebba23cf267942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6181b166b5cb429da3ebba23cf267942 already in progress: creating tablet
W20250112 20:51:51.355223 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9b0578685a045c0801e875f8f42cb0f consensus state has no leader
W20250112 20:51:51.355520 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b95e22f892d24fe390619250ef5fba8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.356072 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8d0fe81bfac94ece91fa136594e0ae88" } new_table_name: "default.Table_999071fc1abc4f929b60eaaae270a883" modify_external_catalogs: true
I20250112 20:51:51.356842 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet b496462873e44ceab2ec3c95cf63fc5a
I20250112 20:51:51.357311 446 catalog_manager.cc:6596] ReplaceTablet: tablet b496462873e44ceab2ec3c95cf63fc5a deleted and replaced by tablet 86b51849dd2647508ec6ec06503ff850
I20250112 20:51:51.356700 455 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:51.358502 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_5147d452317249dd9a587356732a3d59 [id=e8f4d732342e43bda26b51c8e1122d80] alter complete (version 2)
W20250112 20:51:51.359201 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:36661 (request call id 181) took 853 ms (client timeout 1000 ms). Trace:
I20250112 20:51:51.359449 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d571de1e2b104ba480cdaf2523b54599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:51.359295 452 rpcz_store.cc:269] 0112 20:51:50.505667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.577573 (+ 71906us) service_pool.cc:225] Handling call
0112 20:51:50.581032 (+ 3459us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x616000041b54. stack: 00007fa2ddff5fd6 00007fa2ddff5d90 00007fa2dcfd548d 000000000096d544 000000000096d50d 00007fa2ecb93e1d 00007fa2ecb93f13 00007fa2e989c211 00007fa2f13739dd 00007fa2f14f96b8 00007fa2e85fe716 00007fa2e85fe445 00007fa2e4d5d30e 00007fa2e4d5c1ee 00007fa2e4d62425 00007fa2e4d62d4f
0112 20:51:51.359141 (+778109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:51.086439 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 20:51:51.089704 (+ 3265us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:51.089711 (+ 7us) write_op.cc:435] Acquired schema lock
0112 20:51:51.089713 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:51.089833 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:51.089845 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:51.089850 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:51.089922 (+ 72us) tablet.cc:698] Row locks acquired
0112 20:51:51.089924 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:51.089957 (+ 33us) write_op.cc:270] Start()
0112 20:51:51.089988 (+ 31us) write_op.cc:276] Timestamp: P: 1736715111089910 usec, L: 0
0112 20:51:51.089994 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:51.090183 (+ 189us) log.cc:844] Serialized 356 byte log entry
0112 20:51:51.176292 (+ 86109us) op_driver.cc:464] REPLICATION: finished
0112 20:51:51.176385 (+ 93us) write_op.cc:301] APPLY: starting
0112 20:51:51.176406 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:51.176455 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:51.176485 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:51.176545 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:51.176587 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 20:51:51.176599 (+ 12us) write_op.cc:312] APPLY: finished
0112 20:51:51.176689 (+ 90us) log.cc:844] Serialized 23 byte log entry
0112 20:51:51.176757 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:51.176779 (+ 22us) write_op.cc:454] Released schema lock
0112 20:51:51.176789 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":6186240,"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":503,"apply.run_wall_time_us":503,"num_ops":1,"prepare.queue_time_us":206150,"prepare.run_cpu_time_us":791,"prepare.run_wall_time_us":50230,"raft.queue_time_us":46,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"replication_time_us":86256}]]}
I20250112 20:51:51.359863 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d571de1e2b104ba480cdaf2523b54599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.360102 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1dff4cb9aa364c87b97f2d05462b70ac" } modify_external_catalogs: true
I20250112 20:51:51.360338 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1dff4cb9aa364c87b97f2d05462b70ac" } modify_external_catalogs: true
I20250112 20:51:51.363139 451 catalog_manager.cc:5526] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:51.364082 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:51.364413 451 catalog_manager.cc:5526] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:51.365011 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cb2b5abd46bc4877887604a94159e418" } modify_external_catalogs: true
I20250112 20:51:51.365190 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_cb2b5abd46bc4877887604a94159e418" } modify_external_catalogs: true
I20250112 20:51:51.366245 1163 tablet_bootstrap.cc:654] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.366709 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_826881febc214de081501f6ecb406972" } modify_external_catalogs: true
I20250112 20:51:51.366956 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_826881febc214de081501f6ecb406972" } modify_external_catalogs: true
I20250112 20:51:51.368181 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f98e8647f21b468c96340e5662c7d5f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.368686 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f98e8647f21b468c96340e5662c7d5f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.369124 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_292dce92952a4a3ba27a7bb1ed129e3e" } modify_external_catalogs: true
I20250112 20:51:51.369215 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d571de1e2b104ba480cdaf2523b54599 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:50 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.370065 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_89746631f84f4bb097908ead01455c16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.370296 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_292dce92952a4a3ba27a7bb1ed129e3e" } modify_external_catalogs: true
I20250112 20:51:51.370631 3284 raft_consensus.cc:491] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:51.370726 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89746631f84f4bb097908ead01455c16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.370759 3284 raft_consensus.cc:513] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.371353 3284 leader_election.cc:290] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:51.371845 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.378016 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:51.378356 451 catalog_manager.cc:5526] T 0946fe64150e41da957907aa4a6a672f P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:51.378892 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:51.379371 451 catalog_manager.cc:5526] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:51.379835 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:51.389479 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7209914d98d7435883751f90fcdce586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.390077 1164 tablet_bootstrap.cc:654] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:51.390129 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7209914d98d7435883751f90fcdce586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.396950 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb09123a64e744c89cb8343fc27d8d60" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:51.397212 907 raft_consensus.cc:2388] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate aefe1db8b19244d891b6bcd8971e9b17 in this term.
I20250112 20:51:51.400125 3304 raft_consensus.cc:491] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.400283 3304 raft_consensus.cc:513] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.400938 3304 leader_election.cc:290] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.401991 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b00c49f234dc89d63f644febc23fb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.402114 983 leader_election.cc:304] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:51.402165 758 raft_consensus.cc:2463] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:51.402652 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:51.402840 3284 raft_consensus.cc:3054] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.405323 3284 raft_consensus.cc:2743] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.405740 847 leader_election.cc:304] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:51.406392 3304 raft_consensus.cc:2798] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:51.406471 3304 raft_consensus.cc:491] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:51.406530 3304 raft_consensus.cc:3054] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.408645 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1107) took 1102 ms. Trace:
I20250112 20:51:51.408720 1186 rpcz_store.cc:276] 0112 20:51:50.305708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.307389 (+ 1681us) service_pool.cc:225] Handling call
0112 20:51:51.325595 (+1018206us) ts_tablet_manager.cc:1145] Deleting tablet db9178cf782d4bd28eba90cc03796885
0112 20:51:51.325603 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.333653 (+ 8050us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.336297 (+ 2644us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.408630 (+ 72333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018033}
I20250112 20:51:51.409198 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b00c49f234dc89d63f644febc23fb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:51.409263 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57104 (request call id 1109) took 1063 ms. Trace:
I20250112 20:51:51.409339 1186 rpcz_store.cc:276] 0112 20:51:50.403936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.403984 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:51.409005 (+1005021us) ts_tablet_manager.cc:1145] Deleting tablet f7b722b0eb5944a4b344a3833905a2b0
0112 20:51:51.409130 (+ 125us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.409248 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004841}
I20250112 20:51:51.409679 1186 tablet_replica.cc:331] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.409848 1186 raft_consensus.cc:2238] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.409977 1186 raft_consensus.cc:2267] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:51.410295 847 leader_election.cc:343] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.409876 3304 raft_consensus.cc:513] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.410848 1186 ts_tablet_manager.cc:1905] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.411360 1178 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.412791 1178 tablet_bootstrap.cc:654] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.414243 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b00c49f234dc89d63f644febc23fb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:51.414404 758 raft_consensus.cc:3054] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.414506 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b00c49f234dc89d63f644febc23fb" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:51.415011 847 leader_election.cc:343] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.415241 3304 leader_election.cc:290] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:51.416462 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bf2d3a337ce40a99daafa94e159ab8c consensus state has no leader
I20250112 20:51:51.416895 758 raft_consensus.cc:2463] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.417884 847 leader_election.cc:304] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.418668 3304 raft_consensus.cc:2798] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.418782 3304 raft_consensus.cc:695] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:51.419026 3304 consensus_queue.cc:237] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.421229 1186 ts_tablet_manager.cc:1918] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:51.421334 1186 log.cc:1198] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/22f6b2c4ea5f4167896c78b1c64cbb62
I20250112 20:51:51.421633 1186 ts_tablet_manager.cc:1939] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.336231 1185 rpcz_store.cc:276] 0112 20:51:49.079276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.079356 (+ 80us) service_pool.cc:225] Handling call
0112 20:51:51.334458 (+2255102us) ts_tablet_manager.cc:1145] Deleting tablet 0866871b9e1542e8ab30505c68f7dea9
0112 20:51:51.334466 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.334600 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254911}
I20250112 20:51:51.423683 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1045) took 2225 ms. Trace:
I20250112 20:51:51.423820 1185 rpcz_store.cc:276] 0112 20:51:49.198019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.198084 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:51.423541 (+2225457us) ts_tablet_manager.cc:1145] Deleting tablet 532ccaeaf38c41a6ab26f65411d31418
0112 20:51:51.423550 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.423666 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225227}
I20250112 20:51:51.424583 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1047) took 2145 ms. Trace:
I20250112 20:51:51.424724 1185 rpcz_store.cc:276] 0112 20:51:49.279635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.279689 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:51.424480 (+2144791us) ts_tablet_manager.cc:1145] Deleting tablet 01367a11dea04b33a9255508501b9060
0112 20:51:51.424492 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.424566 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144493}
I20250112 20:51:51.425279 1185 tablet_replica.cc:331] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.425580 1185 raft_consensus.cc:2238] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.425788 1185 pending_rounds.cc:62] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:51.425882 1185 pending_rounds.cc:69] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585087054479360 op_type: NO_OP noop_request { }
I20250112 20:51:51.425968 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 480d49569886408a8ce781a93bb5d10d
I20250112 20:51:51.426116 1185 raft_consensus.cc:2883] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:51.426276 1185 raft_consensus.cc:2267] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.426985 1185 ts_tablet_manager.cc:1905] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.427685 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet a2ff6b3dbc614785a5db4b4185fe0945
I20250112 20:51:51.428972 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 008d69d199d64bdf85da17d8d3bf6264
I20250112 20:51:51.433573 1164 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.433795 1164 ts_tablet_manager.cc:1397] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.110s user 0.001s sys 0.003s
I20250112 20:51:51.434612 1164 raft_consensus.cc:357] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.434939 1164 raft_consensus.cc:383] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.435050 1164 raft_consensus.cc:738] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.435384 1164 consensus_queue.cc:260] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.439317 891 tablet_service.cc:1467] Processing CreateTablet for tablet d30b3b402db0475e9c486cba471adea6 (DEFAULT_TABLE table=default.Table_1a006bd6186c4ed89c0a6510ab5ff525 [id=068bdb2acc4f4d9d9f1f66ff180970e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.439692 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30b3b402db0475e9c486cba471adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.439762 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d30b3b402db0475e9c486cba471adea6 (DEFAULT_TABLE table=default.Table_1a006bd6186c4ed89c0a6510ab5ff525 [id=068bdb2acc4f4d9d9f1f66ff180970e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.440110 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30b3b402db0475e9c486cba471adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.452239 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 75790518579d496fb0efb12b497fcc75
I20250112 20:51:51.456216 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5147d452317249dd9a587356732a3d59" } new_table_name: "default.Table_ad1a96f6f9084d4f8daf2c25b1aaec4a" modify_external_catalogs: true
I20250112 20:51:51.456246 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6da9b6b44a4440dd8378e22ca2f174ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.456808 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da9b6b44a4440dd8378e22ca2f174ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.462716 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:51.465310 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:51.466306 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:51.469213 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_44a0d2b28ca7481881dcb3f0efdbb92a [id=462ba020c7414465b636c07442083e81] alter complete (version 2)
I20250112 20:51:51.471911 891 tablet_service.cc:1467] Processing CreateTablet for tablet ae660dcb9c154e44b45701327f21c432 (DEFAULT_TABLE table=default.Table_c544fe5495bb4037970dfebccbbe09b5 [id=9f5af52f896c44979a6a26d9ffd68152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.472177 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ae660dcb9c154e44b45701327f21c432 (DEFAULT_TABLE table=default.Table_c544fe5495bb4037970dfebccbbe09b5 [id=9f5af52f896c44979a6a26d9ffd68152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.472321 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae660dcb9c154e44b45701327f21c432. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.472548 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae660dcb9c154e44b45701327f21c432. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.472945 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_31c3d595c24b47dcb822727869bf8428 [id=f83f83db15594cd9928fccf547cad7aa] alter complete (version 2)
I20250112 20:51:51.476346 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_2c9c479263b44516bafb19e7cd06c176 [id=05211d8527144710860dc2aca1ac76a3] alter complete (version 2)
I20250112 20:51:51.481582 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_f257ef99d9ec49d99f2b4df753f21d4e [id=27536d1722784720b208bbc108bdc02a] alter complete (version 4)
I20250112 20:51:51.483644 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9db38abce01448dabb432c5b84b46dff (DEFAULT_TABLE table=default.Table_ec6d004e89864f5bb239592c66002af8 [id=41cbe4ed7e2a45ebaf3720c40066a3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.484030 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db38abce01448dabb432c5b84b46dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.483984 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9db38abce01448dabb432c5b84b46dff (DEFAULT_TABLE table=default.Table_ec6d004e89864f5bb239592c66002af8 [id=41cbe4ed7e2a45ebaf3720c40066a3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.484411 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db38abce01448dabb432c5b84b46dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.485105 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3014e9524fe0bb367e81226dc022 (DEFAULT_TABLE table=default.Table_c36b678fdc7544dc9acc25643ccab97c [id=2b6e8a2149524b708e1351f4fecadcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.485399 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef3014e9524fe0bb367e81226dc022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.483292 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_418cbc0f13b942718efd3ac122db95ca [id=7ab67c588df54351a13701edb4b072c3] alter complete (version 1)
I20250112 20:51:51.489051 889 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3014e9524fe0bb367e81226dc022 (DEFAULT_TABLE table=default.Table_c36b678fdc7544dc9acc25643ccab97c [id=2b6e8a2149524b708e1351f4fecadcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.489090 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 20c630deb4dd4b5ba541a9e572a8c320 (DEFAULT_TABLE table=default.Table_b043070379134d4bb916ee883c48fccd [id=3931c403e0404b02a688c602c291274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.489387 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef3014e9524fe0bb367e81226dc022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.489503 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c630deb4dd4b5ba541a9e572a8c320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.491724 1163 tablet_bootstrap.cc:492] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.491832 1163 ts_tablet_manager.cc:1397] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.205s user 0.003s sys 0.000s
I20250112 20:51:51.492506 1163 raft_consensus.cc:357] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.492677 1163 raft_consensus.cc:383] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.492722 1163 raft_consensus.cc:738] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.493016 884 tablet_service.cc:1467] Processing CreateTablet for tablet 20c630deb4dd4b5ba541a9e572a8c320 (DEFAULT_TABLE table=default.Table_b043070379134d4bb916ee883c48fccd [id=3931c403e0404b02a688c602c291274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.492955 1163 consensus_queue.cc:260] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.493314 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c630deb4dd4b5ba541a9e572a8c320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.496270 1163 ts_tablet_manager.cc:1428] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:51.496522 1163 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:51.497907 1163 tablet_bootstrap.cc:654] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.501556 3268 raft_consensus.cc:491] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.501693 3268 raft_consensus.cc:513] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.502075 756 tablet_service.cc:1467] Processing CreateTablet for tablet d30b3b402db0475e9c486cba471adea6 (DEFAULT_TABLE table=default.Table_1a006bd6186c4ed89c0a6510ab5ff525 [id=068bdb2acc4f4d9d9f1f66ff180970e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.502189 742 tablet_service.cc:1467] Processing CreateTablet for tablet ae660dcb9c154e44b45701327f21c432 (DEFAULT_TABLE table=default.Table_c544fe5495bb4037970dfebccbbe09b5 [id=9f5af52f896c44979a6a26d9ffd68152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.502346 3268 leader_election.cc:290] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:51.502406 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30b3b402db0475e9c486cba471adea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.503245 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9db38abce01448dabb432c5b84b46dff (DEFAULT_TABLE table=default.Table_ec6d004e89864f5bb239592c66002af8 [id=41cbe4ed7e2a45ebaf3720c40066a3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.503547 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db38abce01448dabb432c5b84b46dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.504982 750 tablet_service.cc:1467] Processing CreateTablet for tablet 41ef3014e9524fe0bb367e81226dc022 (DEFAULT_TABLE table=default.Table_c36b678fdc7544dc9acc25643ccab97c [id=2b6e8a2149524b708e1351f4fecadcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.505270 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41ef3014e9524fe0bb367e81226dc022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.508919 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:51.509601 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:51.510664 710 leader_election.cc:343] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:51.510932 710 leader_election.cc:343] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.511021 710 leader_election.cc:304] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:51.511529 3121 raft_consensus.cc:2743] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.513367 1164 ts_tablet_manager.cc:1428] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20250112 20:51:51.513690 1164 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.515105 1164 tablet_bootstrap.cc:654] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.516721 889 tablet_service.cc:1467] Processing CreateTablet for tablet fd8afd190cba44e28f96deb5f9284647 (DEFAULT_TABLE table=default.Table_e2436c1c8e5b4795a868b7f537315d19 [id=fbe6f9716cc946949dd9fa328dfd738b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.516983 750 tablet_service.cc:1467] Processing CreateTablet for tablet 20c630deb4dd4b5ba541a9e572a8c320 (DEFAULT_TABLE table=default.Table_b043070379134d4bb916ee883c48fccd [id=3931c403e0404b02a688c602c291274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.516933 1014 tablet_service.cc:1467] Processing CreateTablet for tablet fd8afd190cba44e28f96deb5f9284647 (DEFAULT_TABLE table=default.Table_e2436c1c8e5b4795a868b7f537315d19 [id=fbe6f9716cc946949dd9fa328dfd738b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.517416 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c630deb4dd4b5ba541a9e572a8c320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.517613 1010 tablet_service.cc:1467] Processing CreateTablet for tablet f84396f2bf414bac8fc03664e0cb4579 (DEFAULT_TABLE table=default.Table_c1535f7854fa43009733487f7b3af237 [id=143f79c99a4c44089ed516d6e3dc8118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.518740 751 tablet_service.cc:1467] Processing CreateTablet for tablet f84396f2bf414bac8fc03664e0cb4579 (DEFAULT_TABLE table=default.Table_c1535f7854fa43009733487f7b3af237 [id=143f79c99a4c44089ed516d6e3dc8118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.519169 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84396f2bf414bac8fc03664e0cb4579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.521219 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8afd190cba44e28f96deb5f9284647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.521360 756 tablet_service.cc:1467] Processing CreateTablet for tablet fd8afd190cba44e28f96deb5f9284647 (DEFAULT_TABLE table=default.Table_e2436c1c8e5b4795a868b7f537315d19 [id=fbe6f9716cc946949dd9fa328dfd738b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.521732 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8afd190cba44e28f96deb5f9284647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.521823 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84396f2bf414bac8fc03664e0cb4579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.521031 884 tablet_service.cc:1467] Processing CreateTablet for tablet f84396f2bf414bac8fc03664e0cb4579 (DEFAULT_TABLE table=default.Table_c1535f7854fa43009733487f7b3af237 [id=143f79c99a4c44089ed516d6e3dc8118]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.525015 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84396f2bf414bac8fc03664e0cb4579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.531288 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8afd190cba44e28f96deb5f9284647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.540050 2447 ts_tablet_manager.cc:1918] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:51.540225 2447 log.cc:1198] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1fd47d1c1a4b42bdbff97f619dc95134
I20250112 20:51:51.540580 2447 ts_tablet_manager.cc:1939] T 1fd47d1c1a4b42bdbff97f619dc95134 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.541508 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 975) took 3048 ms. Trace:
I20250112 20:51:51.541656 1178 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.541649 2447 rpcz_store.cc:276] 0112 20:51:48.493481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.493650 (+ 169us) service_pool.cc:225] Handling call
0112 20:51:51.296229 (+2802579us) ts_tablet_manager.cc:1145] Deleting tablet 1fd47d1c1a4b42bdbff97f619dc95134
0112 20:51:51.296237 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.415804 (+119567us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.539790 (+123986us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.541490 (+ 1700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2802090}
I20250112 20:51:51.541822 1178 ts_tablet_manager.cc:1397] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.000s
I20250112 20:51:51.542194 2447 tablet_replica.cc:331] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.542415 2447 raft_consensus.cc:2238] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.542734 2447 raft_consensus.cc:2267] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.543306 1178 raft_consensus.cc:357] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.544041 1178 raft_consensus.cc:383] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.544152 1178 raft_consensus.cc:738] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.544401 1178 consensus_queue.cc:260] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.545114 1178 ts_tablet_manager.cc:1428] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:51.550475 3304 raft_consensus.cc:491] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.555781 3304 raft_consensus.cc:513] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.556363 3304 leader_election.cc:290] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.557524 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65784493c6ef417993b1f6371cab47d6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:51.558109 847 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.558531 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b496462873e44ceab2ec3c95cf63fc5a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b51849dd2647508ec6ec06503ff850) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.558578 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 669e1cfd407e4d87a63e6832d9c6fa1f (table default.Table_1a635c00d86942f2a0bbd3dfd848b4d4 [id=c2b4a0a001bf48ec9864c5c48af1df07]) successfully deleted
I20250112 20:51:51.558804 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 924082716c1e4d9ea6fbe97ae93232bd (table default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]) successfully deleted
I20250112 20:51:51.558987 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet dc3a9740415140fe8864d54f477af954 (table default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]) successfully deleted
I20250112 20:51:51.559219 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet c8d24bccf682483aa9b273fc1ce847da (table default.Table_1b743410da4942a091143beef17273fe [id=9351d5e9ae6d45eba7d3bdb7319abd63]) successfully deleted
I20250112 20:51:51.559405 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b496462873e44ceab2ec3c95cf63fc5a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b51849dd2647508ec6ec06503ff850) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.559423 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1fd34e829d0645ea80daaff5d27cb266 (table default.Table_70e7a6a54ee6434988d6cfb21ade1f02 [id=d8b1d14e0b6e430283c7ceed297c1b92]) successfully deleted
W20250112 20:51:51.559610 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet db9bab2477a94c169817d9de8aaad1c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db9bab2477a94c169817d9de8aaad1c0 already in progress: creating tablet
W20250112 20:51:51.559849 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3ea9425f3e7b46288c81dc8cdade99b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ea9425f3e7b46288c81dc8cdade99b2 already in progress: creating tablet
I20250112 20:51:51.559903 1178 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.560313 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1fd47d1c1a4b42bdbff97f619dc95134 (table default.Table_f89fdf37a8944ee0be9e90eec745a613 [id=1b1bf8b2fc0b4fca86f3b29074d33215]) successfully deleted
I20250112 20:51:51.560662 889 tablet_service.cc:1467] Processing CreateTablet for tablet 54e777a247ea46bebff6d099c046b22d (DEFAULT_TABLE table=default.Table_c137b3886d444880aee556714375e8d0 [id=23c5676f3aca42ef948534f45570929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.561076 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e777a247ea46bebff6d099c046b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.563088 1178 tablet_bootstrap.cc:654] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.567431 3325 tablet.cc:1721] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.578199 3097 tablet.cc:1721] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.590718 1164 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.590880 1164 ts_tablet_manager.cc:1397] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.001s
I20250112 20:51:51.591770 1164 raft_consensus.cc:357] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.592011 1164 raft_consensus.cc:383] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.592060 1164 raft_consensus.cc:738] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.595470 3304 raft_consensus.cc:491] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.595623 3304 raft_consensus.cc:513] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.596220 3304 leader_election.cc:290] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.597313 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.597517 758 raft_consensus.cc:2463] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:51.598228 847 leader_election.cc:304] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:51.598582 3304 raft_consensus.cc:2798] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:51.598650 3304 raft_consensus.cc:491] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:51.598727 3304 raft_consensus.cc:3054] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.601970 3304 raft_consensus.cc:513] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.602880 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:51.603057 758 raft_consensus.cc:3054] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.568142 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_8cb2067a639c442685fda6b357d895f6 [id=e4d8998e4e67471d9951cc9dd7ae5284] alter complete (version 1)
W20250112 20:51:51.605437 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 45099.6us
I20250112 20:51:51.605715 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 924082716c1e4d9ea6fbe97ae93232bd (table default.Table_ae1eb78e32524283a93e97294e33bf5a [id=c618f3455b4e47a78b14f0e474c8232e]) successfully deleted
I20250112 20:51:51.606042 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet dc3a9740415140fe8864d54f477af954 (table default.Table_31360e3161b04902a56502dbe4608f86 [id=b3d49324937a4af4b6b2d939e1643bbd]) successfully deleted
I20250112 20:51:51.606292 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 7f807071e53a4f0e8629b1dfa6d26b9d (table default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]) successfully deleted
W20250112 20:51:51.606564 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3ea9425f3e7b46288c81dc8cdade99b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ea9425f3e7b46288c81dc8cdade99b2 already in progress: creating tablet
I20250112 20:51:51.607519 3304 leader_election.cc:290] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.607774 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 45484f8e48524b86a6f418b88a71ac18 (table default.Table_fbe3a476695f40ec8737a0e3e4ad731f [id=eba58b5d38e8443e842cca705dc51aeb]) successfully deleted
I20250112 20:51:51.557125 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65784493c6ef417993b1f6371cab47d6" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:51.566522 983 consensus_peers.cc:487] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0fd4177f9394a5a82d4f5e02369b3b6. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.565806 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b496462873e44ceab2ec3c95cf63fc5a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b51849dd2647508ec6ec06503ff850) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.568807 3250 tablet.cc:1721] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.597827 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:51.607385 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:51.611514 847 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.611624 847 leader_election.cc:304] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:51.611943 847 leader_election.cc:343] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:51.612216 847 leader_election.cc:343] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.605908 758 raft_consensus.cc:2463] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.612773 847 leader_election.cc:304] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.613157 3304 raft_consensus.cc:2743] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.613335 3304 raft_consensus.cc:2798] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.613461 3304 raft_consensus.cc:695] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:51.613684 3304 consensus_queue.cc:237] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.604955 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae660dcb9c154e44b45701327f21c432. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.617765 2447 ts_tablet_manager.cc:1905] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.618125 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f] alter complete (version 1)
W20250112 20:51:51.619313 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet dfc63eff44414906a027a8208b5cc5a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc63eff44414906a027a8208b5cc5a9 already in progress: creating tablet
W20250112 20:51:51.619568 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ed925a5ef3a54f819ca292a4e7967b33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed925a5ef3a54f819ca292a4e7967b33 already in progress: creating tablet
I20250112 20:51:51.619614 1178 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.619728 1178 ts_tablet_manager.cc:1397] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.005s
W20250112 20:51:51.619801 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 3846863c74f64923bda6e5a19e7f2770 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3846863c74f64923bda6e5a19e7f2770 already in progress: creating tablet
W20250112 20:51:51.620004 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 0866871b9e1542e8ab30505c68f7dea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0866871b9e1542e8ab30505c68f7dea9 already in progress: creating tablet
W20250112 20:51:51.620221 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 532ccaeaf38c41a6ab26f65411d31418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ccaeaf38c41a6ab26f65411d31418 already in progress: creating tablet
W20250112 20:51:51.620393 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 01367a11dea04b33a9255508501b9060 because the tablet was not found. No further retry: Not found: Tablet not found: 01367a11dea04b33a9255508501b9060
I20250112 20:51:51.620441 1178 raft_consensus.cc:357] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.620620 1178 raft_consensus.cc:383] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.620676 1178 raft_consensus.cc:738] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.621724 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 85683e97dae44399b1dc7bbd54d62eb6) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.622699 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 480d49569886408a8ce781a93bb5d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.622939 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 480d49569886408a8ce781a93bb5d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.622972 1178 consensus_queue.cc:260] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.623473 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 480d49569886408a8ce781a93bb5d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.624169 1163 tablet_bootstrap.cc:492] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.624331 1163 ts_tablet_manager.cc:1397] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.000s
I20250112 20:51:51.624990 1163 raft_consensus.cc:357] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.623148 1186 tablet_replica.cc:331] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.625260 1163 raft_consensus.cc:383] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.625368 1163 raft_consensus.cc:738] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.625468 1186 raft_consensus.cc:2238] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.625530 1178 ts_tablet_manager.cc:1428] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:51.625631 1186 raft_consensus.cc:2267] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.625626 1163 consensus_queue.cc:260] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.625900 1178 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.626148 1186 ts_tablet_manager.cc:1905] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.626175 1163 ts_tablet_manager.cc:1428] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:51.627528 1178 tablet_bootstrap.cc:654] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.608036 1164 consensus_queue.cc:260] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.630329 1164 ts_tablet_manager.cc:1428] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250112 20:51:51.639377 1186 ts_tablet_manager.cc:1918] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:51.639489 1186 log.cc:1198] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/480d49569886408a8ce781a93bb5d10d
I20250112 20:51:51.639825 1186 ts_tablet_manager.cc:1939] T 480d49569886408a8ce781a93bb5d10d P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.646363 1178 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.646687 1178 ts_tablet_manager.cc:1397] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20250112 20:51:51.648171 1178 raft_consensus.cc:357] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.648548 1178 raft_consensus.cc:383] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.648710 1178 raft_consensus.cc:738] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.649211 1178 consensus_queue.cc:260] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.649791 1164 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.650331 1178 ts_tablet_manager.cc:1428] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:51.651365 1178 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.653975 1164 tablet_bootstrap.cc:654] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.657650 1178 tablet_bootstrap.cc:654] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.661785 1163 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:51.663475 1163 tablet_bootstrap.cc:654] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.673122 891 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.673650 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.673756 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ff6b3dbc614785a5db4b4185fe0945 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.674934 1178 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.675119 1178 ts_tablet_manager.cc:1397] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20250112 20:51:51.676501 1178 raft_consensus.cc:357] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.676800 1178 raft_consensus.cc:383] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.676890 1178 raft_consensus.cc:738] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.677240 1178 consensus_queue.cc:260] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:51.677985 710 consensus_peers.cc:487] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db9178cf782d4bd28eba90cc03796885. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.678213 1178 ts_tablet_manager.cc:1428] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:51.679546 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:51.680100 1185 ts_tablet_manager.cc:1918] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.680226 1185 log.cc:1198] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/1ca8fc4a45fe448b919ac001a7980deb
I20250112 20:51:51.680603 1185 ts_tablet_manager.cc:1939] T 1ca8fc4a45fe448b919ac001a7980deb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:51.681424 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1048) took 2401 ms. Trace:
I20250112 20:51:51.681540 1185 rpcz_store.cc:276] 0112 20:51:49.279985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.280124 (+ 139us) service_pool.cc:225] Handling call
0112 20:51:51.425224 (+2145100us) ts_tablet_manager.cc:1145] Deleting tablet 1ca8fc4a45fe448b919ac001a7980deb
0112 20:51:51.425236 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.545102 (+119866us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.679992 (+134890us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.681407 (+ 1415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144899}
I20250112 20:51:51.681886 1185 tablet_replica.cc:331] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:51.682098 1185 raft_consensus.cc:2238] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.682276 1185 raft_consensus.cc:2267] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.682821 1185 ts_tablet_manager.cc:1905] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.685549 3265 consensus_queue.cc:1035] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.686242 1164 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.686419 1164 ts_tablet_manager.cc:1397] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20250112 20:51:51.699187 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 75790518579d496fb0efb12b497fcc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.699366 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.699448 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.699596 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 75790518579d496fb0efb12b497fcc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.701537 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 75790518579d496fb0efb12b497fcc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.702029 742 tablet_service.cc:1467] Processing CreateTablet for tablet 54e777a247ea46bebff6d099c046b22d (DEFAULT_TABLE table=default.Table_c137b3886d444880aee556714375e8d0 [id=23c5676f3aca42ef948534f45570929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.702337 751 tablet_service.cc:1467] Processing CreateTablet for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d (DEFAULT_TABLE table=default.Table_2050ea3fd15848d7b59a8b8020368824 [id=07f1a598f01e4913b4f2c031c5f9b339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.702989 891 tablet_service.cc:1467] Processing CreateTablet for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d (DEFAULT_TABLE table=default.Table_2050ea3fd15848d7b59a8b8020368824 [id=07f1a598f01e4913b4f2c031c5f9b339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.703346 750 tablet_service.cc:1467] Processing CreateTablet for tablet 03769dfa52b54fabb0f05c17dcbeabe1 (DEFAULT_TABLE table=default.Table_eefdbb79b5994581b1e544fa46ce498b [id=3357f88024a64832bbc1d27a1c467e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.704222 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 669e1cfd407e4d87a63e6832d9c6fa1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:34 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.704203 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 54e777a247ea46bebff6d099c046b22d (DEFAULT_TABLE table=default.Table_c137b3886d444880aee556714375e8d0 [id=23c5676f3aca42ef948534f45570929c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.704360 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 03769dfa52b54fabb0f05c17dcbeabe1 (DEFAULT_TABLE table=default.Table_eefdbb79b5994581b1e544fa46ce498b [id=3357f88024a64832bbc1d27a1c467e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.704424 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.704559 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e777a247ea46bebff6d099c046b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.705049 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03769dfa52b54fabb0f05c17dcbeabe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.707541 1178 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.709807 1178 tablet_bootstrap.cc:654] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.710816 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d (DEFAULT_TABLE table=default.Table_2050ea3fd15848d7b59a8b8020368824 [id=07f1a598f01e4913b4f2c031c5f9b339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.711416 884 tablet_service.cc:1467] Processing CreateTablet for tablet 03769dfa52b54fabb0f05c17dcbeabe1 (DEFAULT_TABLE table=default.Table_eefdbb79b5994581b1e544fa46ce498b [id=3357f88024a64832bbc1d27a1c467e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:51.711733 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03769dfa52b54fabb0f05c17dcbeabe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.716977 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.717247 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.717420 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.717552 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.717684 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.717826 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.719074 1164 raft_consensus.cc:357] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.719415 1164 raft_consensus.cc:383] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.719491 1164 raft_consensus.cc:738] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.719781 1164 consensus_queue.cc:260] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.721333 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.702407 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e777a247ea46bebff6d099c046b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.728752 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6b2c4ea5f4167896c78b1c64cbb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.728973 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03769dfa52b54fabb0f05c17dcbeabe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.729805 1178 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:51.729938 1178 ts_tablet_manager.cc:1397] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20250112 20:51:51.730499 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:51.730664 1178 raft_consensus.cc:357] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.730841 1178 raft_consensus.cc:383] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.730906 1178 raft_consensus.cc:738] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.731130 1178 consensus_queue.cc:260] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.731724 1178 ts_tablet_manager.cc:1428] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:51.731968 1178 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:51.733462 1178 tablet_bootstrap.cc:654] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.737136 3305 raft_consensus.cc:491] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.737257 3305 raft_consensus.cc:513] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.737808 3305 leader_election.cc:290] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.738135 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9bab2477a94c169817d9de8aaad1c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:51.738672 847 leader_election.cc:343] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.739529 3114 consensus_queue.cc:1035] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.739722 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9bab2477a94c169817d9de8aaad1c0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:51.740187 983 consensus_peers.cc:487] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ca8fc4a45fe448b919ac001a7980deb. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.742612 3284 raft_consensus.cc:491] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.742735 3284 raft_consensus.cc:513] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:51.743311 3284 leader_election.cc:290] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:51.744680 983 leader_election.cc:343] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d80791d8582446b89adf212d3d6507eb
W20250112 20:51:51.746115 847 leader_election.cc:343] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.746240 847 leader_election.cc:304] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:51.746803 3305 raft_consensus.cc:2743] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.747046 1164 ts_tablet_manager.cc:1428] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20250112 20:51:51.751117 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80791d8582446b89adf212d3d6507eb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.751358 763 raft_consensus.cc:2388] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:51.753774 983 leader_election.cc:304] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:51.754140 3284 raft_consensus.cc:3054] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.754549 1164 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.756028 1164 tablet_bootstrap.cc:654] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.756414 3284 raft_consensus.cc:2743] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:51.757318 1186 tablet_replica.cc:331] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.757539 1186 raft_consensus.cc:2238] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.757712 1186 raft_consensus.cc:2267] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.758010 1163 tablet_bootstrap.cc:492] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.758183 1163 ts_tablet_manager.cc:1397] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20250112 20:51:51.758311 1186 ts_tablet_manager.cc:1905] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.762543 3265 consensus_queue.cc:1035] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.762640 1163 raft_consensus.cc:357] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.768990 1163 raft_consensus.cc:383] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.769129 1163 raft_consensus.cc:738] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.769435 1163 consensus_queue.cc:260] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.770015 1163 ts_tablet_manager.cc:1428] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:51.770507 1163 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:51.772266 1163 tablet_bootstrap.cc:654] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.777627 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]) successfully deleted
W20250112 20:51:51.777992 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f7b722b0eb5944a4b344a3833905a2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b722b0eb5944a4b344a3833905a2b0 already in progress: creating tablet
I20250112 20:51:51.778373 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
W20250112 20:51:51.778704 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 886690f24fa540a6b496c926bc7a2b07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886690f24fa540a6b496c926bc7a2b07 already in progress: creating tablet
W20250112 20:51:51.778980 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 056ed0fd01824bcea53d58251328898c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056ed0fd01824bcea53d58251328898c already in progress: creating tablet
W20250112 20:51:51.779289 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 096ee7d3a66846f9b619d8f2fc046616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 096ee7d3a66846f9b619d8f2fc046616 already in progress: creating tablet
W20250112 20:51:51.779549 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8074b00eda8d434faaff4b98ae923ebe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8074b00eda8d434faaff4b98ae923ebe already in progress: creating tablet
W20250112 20:51:51.779783 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d571de1e2b104ba480cdaf2523b54599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d571de1e2b104ba480cdaf2523b54599 already in progress: creating tablet
W20250112 20:51:51.780835 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet b496462873e44ceab2ec3c95cf63fc5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b496462873e44ceab2ec3c95cf63fc5a already in progress: creating tablet
W20250112 20:51:51.781101 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 because the tablet was not found. No further retry: Not found: Tablet not found: 9989f3dc61df4f75b9bdb85ba3ca3f37
I20250112 20:51:51.782346 1164 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.782511 1164 ts_tablet_manager.cc:1397] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.001s
I20250112 20:51:51.783196 1164 raft_consensus.cc:357] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.783424 1164 raft_consensus.cc:383] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.783505 1164 raft_consensus.cc:738] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.784071 1164 consensus_queue.cc:260] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.797137 1164 ts_tablet_manager.cc:1428] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250112 20:51:51.797629 3121 consensus_queue.cc:1035] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.798427 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 480d49569886408a8ce781a93bb5d10d (table default.Table_cb2b5abd46bc4877887604a94159e418 [id=d46a20ad8b99492395771d94f5dc60fb]) successfully deleted
W20250112 20:51:51.798923 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet a2ff6b3dbc614785a5db4b4185fe0945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff6b3dbc614785a5db4b4185fe0945 already in progress: creating tablet
W20250112 20:51:51.799320 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 75790518579d496fb0efb12b497fcc75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75790518579d496fb0efb12b497fcc75 already in progress: creating tablet
I20250112 20:51:51.800931 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6188282bedce4235988a0602b6addcaa" } new_table_name: "default.Table_c31a2d167f1f4fb2943ca955214f0aee" modify_external_catalogs: true
W20250112 20:51:51.809386 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 544ea9d983794cf69ec68d96ec5e0438 consensus state has no leader
I20250112 20:51:51.813526 1039 raft_consensus.cc:3054] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.817236 1163 tablet_bootstrap.cc:492] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.817642 1163 ts_tablet_manager.cc:1397] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.001s
I20250112 20:51:51.818037 3121 consensus_queue.cc:1035] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.818519 1163 raft_consensus.cc:357] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.818696 1163 raft_consensus.cc:383] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.818742 1163 raft_consensus.cc:738] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:51.819926 710 consensus_peers.cc:487] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9570269368e14b1abeaffdcc741c6f3a. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:51.821988 3304 raft_consensus.cc:491] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.822196 3304 raft_consensus.cc:513] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.822880 3304 leader_election.cc:290] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.823304 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20d85a4be874b0b90c528a631a29aed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:51.824265 847 leader_election.cc:343] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.824482 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 417ada5111454c54a85cc95db5974bf9 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:51.824770 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20d85a4be874b0b90c528a631a29aed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.818918 1163 consensus_queue.cc:260] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.824994 763 raft_consensus.cc:2463] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:51.824962 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_72096e102a794d418cb83a674ddaeab6" } new_table_name: "default.Table_023f2f432ac74440b9036d7a59766489" modify_external_catalogs: true
I20250112 20:51:51.825507 847 leader_election.cc:304] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.825902 3304 raft_consensus.cc:2798] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:51.826052 3304 raft_consensus.cc:491] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:51.825973 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9647534e6db94ff79224863b3f8ae645"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.826170 3304 raft_consensus.cc:3054] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:51.826655 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9647534e6db94ff79224863b3f8ae645 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.827433 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ff8fdbb27a140239a8286434feff5a2" } modify_external_catalogs: true
I20250112 20:51:51.827652 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9ff8fdbb27a140239a8286434feff5a2" } modify_external_catalogs: true
I20250112 20:51:51.828610 3304 raft_consensus.cc:513] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.829284 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_59bfceefa0364022aa60152f99bcff6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.829743 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20d85a4be874b0b90c528a631a29aed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
W20250112 20:51:51.829793 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59bfceefa0364022aa60152f99bcff6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:51.830206 847 leader_election.cc:343] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:51.830334 3304 leader_election.cc:290] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.830775 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20d85a4be874b0b90c528a631a29aed" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:51.830950 763 raft_consensus.cc:3054] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:51.832859 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 261fe76c57b14b35957ca43606cfc02f consensus state has no leader
I20250112 20:51:51.833534 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 417ada5111454c54a85cc95db5974bf9
I20250112 20:51:51.833536 763 raft_consensus.cc:2463] T a20d85a4be874b0b90c528a631a29aed P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.833873 847 leader_election.cc:304] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:51.833981 455 catalog_manager.cc:6596] ReplaceTablet: tablet 417ada5111454c54a85cc95db5974bf9 deleted and replaced by tablet ade3a33dfabb42499213ada871ddb7d8
I20250112 20:51:51.834646 3304 raft_consensus.cc:2798] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.834936 3304 raft_consensus.cc:695] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:51.835230 3304 consensus_queue.cc:237] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.839524 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 782a195e68124cd391d0394bcb5ec8b2
I20250112 20:51:51.843624 1163 ts_tablet_manager.cc:1428] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250112 20:51:51.844480 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cfebf90ca7954edfbdc4cec024e66dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.845162 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfebf90ca7954edfbdc4cec024e66dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.846988 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_7ff1c2ec5f8d4ed5ad981b81dfa6cd73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.847576 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff1c2ec5f8d4ed5ad981b81dfa6cd73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.848645 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a29c589a2f7043e1939ce20fa09bc1f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.849223 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a29c589a2f7043e1939ce20fa09bc1f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.850183 1163 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:51.850149 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_9f6a90a2894849e08360909243f6a463" } new_table_name: "default.Table_61e4af0125294009a1138ad54621d9b5" modify_external_catalogs: true
I20250112 20:51:51.850739 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b2c5c4da479a4eef84e80dfba42bfc8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.851272 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2c5c4da479a4eef84e80dfba42bfc8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:51.857223 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51d2a44677aa492782c7c805300ed6a5 consensus state has no leader
I20250112 20:51:51.858484 3305 raft_consensus.cc:491] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:51.858651 3305 raft_consensus.cc:513] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.859247 3305 leader_election.cc:290] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.859742 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:51.859947 1041 raft_consensus.cc:2463] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:51.860374 847 leader_election.cc:304] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:51.860649 3305 raft_consensus.cc:2798] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:51.860752 3305 raft_consensus.cc:491] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:51.860857 3305 raft_consensus.cc:3054] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.861303 1163 tablet_bootstrap.cc:654] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.861760 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:51.861979 763 raft_consensus.cc:2463] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:51.862212 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_632fa444ce9749b6a40c85532cb2e6c6" } new_table_name: "default.Table_5222c53944d24d1697298d1ca8ff618b" modify_external_catalogs: true
I20250112 20:51:51.865098 3305 raft_consensus.cc:513] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.865952 3305 leader_election.cc:290] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:51.866457 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:51.866639 1041 raft_consensus.cc:3054] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:51.868528 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5afc68113174c179d33fac8f46e4de5 consensus state has no leader
I20250112 20:51:51.869400 1041 raft_consensus.cc:2463] T d72ae28879dc45429de2ab0d66276255 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.869791 847 leader_election.cc:304] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:51.870102 3305 raft_consensus.cc:2798] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.870502 3305 raft_consensus.cc:695] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:51.870910 3305 consensus_queue.cc:237] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.873781 1186 ts_tablet_manager.cc:1918] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:51.873889 1186 log.cc:1198] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/d17e6d263014448b96c5d1f03013e6e7
I20250112 20:51:51.874353 1186 ts_tablet_manager.cc:1939] T d17e6d263014448b96c5d1f03013e6e7 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:51.874764 1186 tablet_replica.cc:331] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:51.874977 1186 raft_consensus.cc:2238] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:51.875196 1186 pending_rounds.cc:62] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:51.875384 884 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.875384 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.877718 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 417ada5111454c54a85cc95db5974bf9 with delete_type TABLET_DATA_DELETED (replaced by tablet ade3a33dfabb42499213ada871ddb7d8) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.877743 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 417ada5111454c54a85cc95db5974bf9 with delete_type TABLET_DATA_DELETED (replaced by tablet ade3a33dfabb42499213ada871ddb7d8) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.878497 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.879674 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.880067 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 417ada5111454c54a85cc95db5974bf9 with delete_type TABLET_DATA_DELETED (replaced by tablet ade3a33dfabb42499213ada871ddb7d8) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:51.880537 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 327678us
I20250112 20:51:51.881430 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.884231 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 1ca8fc4a45fe448b919ac001a7980deb (table default.Table_a1454445414a4f2882deedb81bad67bf [id=a40bd70a54ec4854a36809ac58df29b0]) successfully deleted
I20250112 20:51:51.885912 3121 consensus_queue.cc:1035] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.886693 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72ae28879dc45429de2ab0d66276255" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:51.886873 763 raft_consensus.cc:3054] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.889464 763 raft_consensus.cc:2463] T d72ae28879dc45429de2ab0d66276255 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:51.888810 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.893344 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 782a195e68124cd391d0394bcb5ec8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.894923 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_963ba961a67b40e08a07744599f1673d" } modify_external_catalogs: true
I20250112 20:51:51.895149 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_963ba961a67b40e08a07744599f1673d" } modify_external_catalogs: true
I20250112 20:51:51.903192 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet df292cbdf58b4fcd889fcb63335b37d9
I20250112 20:51:51.905787 2447 ts_tablet_manager.cc:1918] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:51.905985 2447 log.cc:1198] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/7f807071e53a4f0e8629b1dfa6d26b9d
I20250112 20:51:51.906445 2447 ts_tablet_manager.cc:1939] T 7f807071e53a4f0e8629b1dfa6d26b9d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:51.907362 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d4025212c78a43f2af864cca35e42251" } modify_external_catalogs: true
I20250112 20:51:51.907563 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d4025212c78a43f2af864cca35e42251" } modify_external_catalogs: true
I20250112 20:51:51.908399 3121 consensus_queue.cc:1035] T 480dca3c3a1e4da7a84834e02f26b299 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:51.912704 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e553dd8387024dbca022b562a69871a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.913405 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e553dd8387024dbca022b562a69871a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.914145 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_89b51b6ef2d74df7a45fb8c00b3b4538" } modify_external_catalogs: true
I20250112 20:51:51.914338 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_89b51b6ef2d74df7a45fb8c00b3b4538" } modify_external_catalogs: true
I20250112 20:51:51.915026 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 91243c9c6deb447e8246f7abe2691f55
I20250112 20:51:51.916322 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_631ecd1223de417bbc68cb50d4805337"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:51.916827 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_631ecd1223de417bbc68cb50d4805337 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:51.917688 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet d17e6d263014448b96c5d1f03013e6e7 (table default.Table_a1b5b3029a0f4295904392987db19062 [id=04bd0f25c5db41039b53f79ce61124ae]) successfully deleted
I20250112 20:51:51.920130 742 tablet_service.cc:1514] Processing DeleteTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.923066 889 tablet_service.cc:1514] Processing DeleteTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.925201 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 91243c9c6deb447e8246f7abe2691f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.927668 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:51.927978 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:51.928417 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:51.928660 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 640dc9187c2243248ef73bac6616c126 (table default.Table_ad1a96f6f9084d4f8daf2c25b1aaec4a [id=e8f4d732342e43bda26b51c8e1122d80]). Expected version 3 got 2
I20250112 20:51:51.931264 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 91243c9c6deb447e8246f7abe2691f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:51.940006 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.942430 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.943213 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.944236 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.954171 3097 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:51.955127 1164 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:51.956601 1164 tablet_bootstrap.cc:654] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:51.959446 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:51.960851 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:51.961110 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:51.965219 1164 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:51.965489 1164 ts_tablet_manager.cc:1397] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250112 20:51:51.875255 1186 pending_rounds.cc:69] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585096379457536 op_type: NO_OP noop_request { }
I20250112 20:51:51.968423 1186 raft_consensus.cc:2883] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:51.968559 1186 raft_consensus.cc:2267] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.968712 1164 raft_consensus.cc:357] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:51.969158 1164 raft_consensus.cc:383] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:51.969307 1164 raft_consensus.cc:738] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.978271 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:51.978353 1186 ts_tablet_manager.cc:1905] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:51.979779 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8f71e61c496a49e18c699f7850924379" } modify_external_catalogs: true
I20250112 20:51:51.979959 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8f71e61c496a49e18c699f7850924379" } modify_external_catalogs: true
I20250112 20:51:51.982208 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:51.983101 450 catalog_manager.cc:5526] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:51.985159 3325 tablet.cc:1721] T fb09123a64e744c89cb8343fc27d8d60 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:51.988523 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:51.988831 450 catalog_manager.cc:5526] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:51.989501 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:52.019325 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:52.023168 3325 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:52.027752 3325 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:52.030515 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet fb09123a64e744c89cb8343fc27d8d60 (table default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda]). Expected version 1 got 0
I20250112 20:51:52.044318 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_ad1a96f6f9084d4f8daf2c25b1aaec4a [id=e8f4d732342e43bda26b51c8e1122d80] alter complete (version 3)
I20250112 20:51:52.054239 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 974) took 3560 ms. Trace:
I20250112 20:51:52.054402 2447 rpcz_store.cc:276] 0112 20:51:48.493410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.494227 (+ 817us) service_pool.cc:225] Handling call
0112 20:51:51.542153 (+3047926us) ts_tablet_manager.cc:1145] Deleting tablet 7f807071e53a4f0e8629b1dfa6d26b9d
0112 20:51:51.542162 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.865309 (+323147us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:51.905674 (+ 40365us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.054217 (+148543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3047816}
I20250112 20:51:52.055094 2447 tablet_replica.cc:331] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:51.923117 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet df292cbdf58b4fcd889fcb63335b37d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.926271 1163 tablet_bootstrap.cc:492] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:51.930037 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 91243c9c6deb447e8246f7abe2691f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.955621 3250 tablet.cc:1721] T 640dc9187c2243248ef73bac6616c126 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:52.055393 1163 ts_tablet_manager.cc:1397] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.205s user 0.003s sys 0.000s
I20250112 20:51:52.055444 2447 raft_consensus.cc:2238] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:51.990823 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d80791d8582446b89adf212d3d6507eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:44 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.055691 2447 raft_consensus.cc:2267] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:51.990967 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:51.991070 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a5162a25ce47519a091742f3d5ca31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.056125 1163 raft_consensus.cc:357] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.056303 1163 raft_consensus.cc:383] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.056349 1163 raft_consensus.cc:738] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.056514 1163 consensus_queue.cc:260] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.056993 1163 ts_tablet_manager.cc:1428] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:52.057245 1163 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:52.057653 2447 ts_tablet_manager.cc:1905] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.058847 1163 tablet_bootstrap.cc:654] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.021720 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d17e6d263014448b96c5d1f03013e6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.036890 3373 raft_consensus.cc:491] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.061177 3373 raft_consensus.cc:513] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.062292 3373 leader_election.cc:290] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.066040 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.066295 907 raft_consensus.cc:2463] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:52.068022 983 leader_election.cc:304] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:52.070292 3373 raft_consensus.cc:2798] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.070959 3373 raft_consensus.cc:491] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.071549 3373 raft_consensus.cc:3054] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.071831 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_9065cbdefe5349b5b6c70898bec1b850"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.073417 448 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 2 got 0
I20250112 20:51:52.074290 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a0aadee20eb34f1eb4f122d7d5920824"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:51.969847 1164 consensus_queue.cc:260] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.077675 1164 ts_tablet_manager.cc:1428] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.112s user 0.002s sys 0.000s
I20250112 20:51:52.077944 1164 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.079495 1164 tablet_bootstrap.cc:654] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:52.088624 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 69556.3us
I20250112 20:51:52.089738 1163 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.089947 1163 ts_tablet_manager.cc:1397] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20250112 20:51:52.090668 1163 raft_consensus.cc:357] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.090956 1163 raft_consensus.cc:383] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.091087 1163 raft_consensus.cc:738] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.091336 3250 tablet.cc:1721] T fb09123a64e744c89cb8343fc27d8d60 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:52.092044 1163 consensus_queue.cc:260] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.092221 3097 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:52.092579 1163 ts_tablet_manager.cc:1428] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:52.092854 1163 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:52.094267 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.094408 1163 tablet_bootstrap.cc:654] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.094540 763 raft_consensus.cc:2463] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
W20250112 20:51:52.095436 710 consensus_peers.cc:487] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22f6b2c4ea5f4167896c78b1c64cbb62. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:52.095861 710 consensus_peers.cc:487] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f264d84ae4846d1afaaabc41d5789f4. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:52.100517 3250 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:52.116791 3097 tablet.cc:1721] T fb09123a64e744c89cb8343fc27d8d60 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:52.123958 1185 ts_tablet_manager.cc:1918] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.124370 1185 log.cc:1198] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9989f3dc61df4f75b9bdb85ba3ca3f37
I20250112 20:51:52.125197 1185 ts_tablet_manager.cc:1939] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.138285 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 23dc8a695ccd4f88bd108304add01a4e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:52.144397 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e45ad07d53b94b6f803b6b694656de77" } new_table_name: "default.Table_485bd35db7f84b538750ddfcbca8eeff" modify_external_catalogs: true
W20250112 20:51:52.146428 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 30321942214949b3bcb727d49f4a34d6 (table default.Table_89b51b6ef2d74df7a45fb8c00b3b4538 [id=38e81f9757ab481f99dbd49ab35e534b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 30321942214949b3bcb727d49f4a34d6 consensus state has no leader
I20250112 20:51:52.151028 3265 raft_consensus.cc:491] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.151178 3265 raft_consensus.cc:513] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.151760 3265 leader_election.cc:290] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.152458 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1049) took 2871 ms. Trace:
I20250112 20:51:52.152710 1185 rpcz_store.cc:276] 0112 20:51:49.280997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.281043 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:51.681843 (+2400800us) ts_tablet_manager.cc:1145] Deleting tablet 9989f3dc61df4f75b9bdb85ba3ca3f37
0112 20:51:51.681853 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:51.999796 (+317943us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.123770 (+123974us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.152438 (+ 28668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400646}
I20250112 20:51:52.152127 3373 raft_consensus.cc:513] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.154528 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1046) took 2875 ms. Trace:
I20250112 20:51:52.154703 1185 rpcz_store.cc:276] 0112 20:51:49.279346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.282984 (+ 3638us) service_pool.cc:225] Handling call
0112 20:51:52.154277 (+2871293us) ts_tablet_manager.cc:1145] Deleting tablet 1d97ac93467a4a7a8ae44d52719f191c
0112 20:51:52.154319 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.154473 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871106}
I20250112 20:51:52.155421 1185 tablet_replica.cc:331] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:52.155864 1185 raft_consensus.cc:2238] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.156319 1185 raft_consensus.cc:2267] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.155893 1164 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.156891 1164 ts_tablet_manager.cc:1397] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.006s
I20250112 20:51:52.157457 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_2b37091fbda64e07b5641542b2961d72 [id=3ddc179c72f74f78b0ec696d624429c4] alter complete (version 2)
I20250112 20:51:52.157775 1164 raft_consensus.cc:357] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.158070 1164 raft_consensus.cc:383] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.158205 1164 raft_consensus.cc:738] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.158677 1164 consensus_queue.cc:260] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.160123 1164 ts_tablet_manager.cc:1428] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250112 20:51:52.161844 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d99f1ae971034e87a92136cc812aa13b consensus state has no leader
I20250112 20:51:52.179234 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_059de69672e845ae99db3f0615692b3f [id=55b0f31b541646faa7a63e7f80a15dda] alter complete (version 1)
I20250112 20:51:52.188838 1186 ts_tablet_manager.cc:1918] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.189072 1186 log.cc:1198] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b37b00c49f234dc89d63f644febc23fb
W20250112 20:51:52.190428 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 89238.5us
I20250112 20:51:52.191043 1185 ts_tablet_manager.cc:1905] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.191303 3121 consensus_queue.cc:1035] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:52.194859 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4fa71cf46144629e1fc1f2172fc3cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:52.195051 1178 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.195173 1178 ts_tablet_manager.cc:1397] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.463s user 0.108s sys 0.178s
I20250112 20:51:52.195901 1178 raft_consensus.cc:357] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.196166 1178 raft_consensus.cc:383] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.197108 3265 raft_consensus.cc:491] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.197223 3265 raft_consensus.cc:513] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.197764 3265 leader_election.cc:290] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.198577 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4fa71cf46144629e1fc1f2172fc3cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.198836 1178 raft_consensus.cc:738] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.199080 1178 consensus_queue.cc:260] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.199638 1178 ts_tablet_manager.cc:1428] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:51:52.200932 1178 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.054883 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 7f807071e53a4f0e8629b1dfa6d26b9d (table default.Table_4283dadebf89422795d890e3991db698 [id=1ac190987d7b4a93b1e8dd4a6fe6ac27]) successfully deleted
I20250112 20:51:52.201936 907 raft_consensus.cc:2463] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:52.205050 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9065cbdefe5349b5b6c70898bec1b850 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:52.206835 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0aadee20eb34f1eb4f122d7d5920824 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.207917 1178 tablet_bootstrap.cc:654] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.208621 1186 ts_tablet_manager.cc:1939] T b37b00c49f234dc89d63f644febc23fb P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.210193 1186 tablet_replica.cc:331] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.210391 1186 raft_consensus.cc:2238] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.210544 1186 raft_consensus.cc:2267] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:52.210884 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 146978us
I20250112 20:51:52.216250 1164 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.217299 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b37b00c49f234dc89d63f644febc23fb (table default.Table_e907fd649a434fe9b3a51b2619964cb3 [id=8997a20696974e4fb75b4f18903d7def]) successfully deleted
I20250112 20:51:52.217938 1164 tablet_bootstrap.cc:654] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.220398 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:52.220371 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.222002 1178 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.222076 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (table default.Table_f020d2e2fdf24b08ba8be0a7cfb00340 [id=84350f6ee4914832aa5cc23c40a9594c]) successfully deleted
W20250112 20:51:52.222359 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 1d97ac93467a4a7a8ae44d52719f191c because the tablet was not found. No further retry: Not found: Tablet not found: 1d97ac93467a4a7a8ae44d52719f191c
I20250112 20:51:52.222462 1178 ts_tablet_manager.cc:1397] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250112 20:51:52.223259 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_bd0bb2087a8b4226ac58760eb0cc949d" } new_table_name: "default.Table_b3604ab4867441d18ff85765a0a1de68" modify_external_catalogs: true
W20250112 20:51:52.224154 710 leader_election.cc:343] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:52.224404 710 leader_election.cc:343] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.224638 1178 raft_consensus.cc:357] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.225174 1178 raft_consensus.cc:383] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.225224 710 leader_election.cc:304] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:52.225390 1178 raft_consensus.cc:738] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:52.225576 710 leader_election.cc:343] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.225677 710 leader_election.cc:304] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:52.225873 1178 consensus_queue.cc:260] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.226043 3114 raft_consensus.cc:2798] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.226125 3114 raft_consensus.cc:491] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.226202 3114 raft_consensus.cc:3054] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.226841 1178 ts_tablet_manager.cc:1428] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 20:51:52.226846 2447 ts_tablet_manager.cc:1918] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:52.227677 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8b22dc96a929414faa15f9131ffc49b1" } new_table_name: "default.Table_cc50dff08c4840e1bc48b28b1dc50ef9" modify_external_catalogs: true
I20250112 20:51:52.229043 3268 raft_consensus.cc:2743] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:52.229709 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b3d4c836e34b461690df48ea31e4617a" } modify_external_catalogs: true
I20250112 20:51:52.229923 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_b3d4c836e34b461690df48ea31e4617a" } modify_external_catalogs: true
I20250112 20:51:52.232122 3114 raft_consensus.cc:513] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.232520 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ec6d004e89864f5bb239592c66002af8" } new_table_name: "default.Table_60265b7c7f4b48b7929462845b27fa6f" modify_external_catalogs: true
I20250112 20:51:52.232892 3114 leader_election.cc:290] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.238462 3373 leader_election.cc:290] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:52.238795 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da93a5d375334552bc68510aebb507dd consensus state has no leader
I20250112 20:51:52.238791 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:52.238978 907 raft_consensus.cc:3054] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.239174 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_da12b2e64bd640d9b17b825b8653b314" } modify_external_catalogs: true
I20250112 20:51:52.239349 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_da12b2e64bd640d9b17b825b8653b314" } modify_external_catalogs: true
W20250112 20:51:52.240015 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d44c14aead8249a291b194f228c5ca2b consensus state has no leader
I20250112 20:51:52.240381 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_69b0567cbb9e451fab36564e4c6582ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.240509 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5a2776514bca4b24ade2fab5a7ed71df" } modify_external_catalogs: true
I20250112 20:51:52.240623 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5a2776514bca4b24ade2fab5a7ed71df" } modify_external_catalogs: true
W20250112 20:51:52.240978 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b0567cbb9e451fab36564e4c6582ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.242285 907 raft_consensus.cc:2463] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:52.242787 983 leader_election.cc:304] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:52.243759 1178 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.244107 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8523098ce690434c99af052e3170ca62" } new_table_name: "default.Table_07ba515d8c374c8b88b45de17632482e" modify_external_catalogs: true
I20250112 20:51:52.245725 1178 tablet_bootstrap.cc:654] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.250751 3178 raft_consensus.cc:491] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.250918 3178 raft_consensus.cc:513] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.251096 1185 ts_tablet_manager.cc:1918] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.251641 3178 leader_election.cc:290] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.252617 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:52.253268 847 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.253922 3373 raft_consensus.cc:2798] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.254199 3373 raft_consensus.cc:695] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:52.254514 3373 consensus_queue.cc:237] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.257678 1178 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.257843 1178 ts_tablet_manager.cc:1397] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
W20250112 20:51:52.257875 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62796226707a4a768bcd699c1260d522 consensus state has no leader
I20250112 20:51:52.251240 1185 log.cc:1198] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/98547168378947df885c601ccdc85e57
W20250112 20:51:52.258422 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9db38abce01448dabb432c5b84b46dff consensus state has no leader
I20250112 20:51:52.226243 1186 ts_tablet_manager.cc:1905] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.258610 1185 ts_tablet_manager.cc:1939] T 98547168378947df885c601ccdc85e57 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.259181 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1050) took 2907 ms. Trace:
I20250112 20:51:52.226939 2447 log.cc:1198] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/45484f8e48524b86a6f418b88a71ac18
I20250112 20:51:52.261386 2447 ts_tablet_manager.cc:1939] T 45484f8e48524b86a6f418b88a71ac18 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.262287 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4fa71cf46144629e1fc1f2172fc3cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:52.265228 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 980) took 3771 ms. Trace:
I20250112 20:51:52.265373 2447 rpcz_store.cc:276] 0112 20:51:48.493939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.495686 (+ 1747us) service_pool.cc:225] Handling call
0112 20:51:52.055025 (+3559339us) ts_tablet_manager.cc:1145] Deleting tablet 45484f8e48524b86a6f418b88a71ac18
0112 20:51:52.055034 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.162771 (+107737us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.226751 (+ 63980us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.265204 (+ 38453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3559184}
I20250112 20:51:52.266084 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 981) took 3589 ms. Trace:
I20250112 20:51:52.266245 2447 rpcz_store.cc:276] 0112 20:51:48.676183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.676228 (+ 45us) service_pool.cc:225] Handling call
0112 20:51:52.265935 (+3589707us) ts_tablet_manager.cc:1145] Deleting tablet dfc63eff44414906a027a8208b5cc5a9
0112 20:51:52.265947 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.266067 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589535}
I20250112 20:51:52.266866 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 982) took 3513 ms. Trace:
I20250112 20:51:52.267063 2447 rpcz_store.cc:276] 0112 20:51:48.753722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.753830 (+ 108us) service_pool.cc:225] Handling call
0112 20:51:52.266746 (+3512916us) ts_tablet_manager.cc:1145] Deleting tablet ed925a5ef3a54f819ca292a4e7967b33
0112 20:51:52.266758 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.266848 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3512666}
I20250112 20:51:52.267959 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 45484f8e48524b86a6f418b88a71ac18 (table default.Table_fbe3a476695f40ec8737a0e3e4ad731f [id=eba58b5d38e8443e842cca705dc51aeb]) successfully deleted
W20250112 20:51:52.268196 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet dfc63eff44414906a027a8208b5cc5a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc63eff44414906a027a8208b5cc5a9 already in progress: creating tablet
W20250112 20:51:52.268371 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ed925a5ef3a54f819ca292a4e7967b33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed925a5ef3a54f819ca292a4e7967b33 already in progress: creating tablet
I20250112 20:51:52.261009 1178 raft_consensus.cc:357] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.268992 1178 raft_consensus.cc:383] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.269095 1178 raft_consensus.cc:738] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.269398 1178 consensus_queue.cc:260] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.270072 1178 ts_tablet_manager.cc:1428] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:52.267678 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 983) took 3433 ms. Trace:
I20250112 20:51:52.271173 2447 rpcz_store.cc:276] 0112 20:51:48.834028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.834114 (+ 86us) service_pool.cc:225] Handling call
0112 20:51:52.267566 (+3433452us) ts_tablet_manager.cc:1145] Deleting tablet 3846863c74f64923bda6e5a19e7f2770
0112 20:51:52.267577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.267663 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3433196}
W20250112 20:51:52.271577 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 3846863c74f64923bda6e5a19e7f2770 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3846863c74f64923bda6e5a19e7f2770 already in progress: creating tablet
I20250112 20:51:52.271643 2447 tablet_replica.cc:331] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.271911 2447 raft_consensus.cc:2238] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.272084 2447 raft_consensus.cc:2267] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.272706 2447 ts_tablet_manager.cc:1905] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.273478 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.259277 1185 rpcz_store.cc:276] 0112 20:51:49.351929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.352024 (+ 95us) service_pool.cc:225] Handling call
0112 20:51:52.155314 (+2803290us) ts_tablet_manager.cc:1145] Deleting tablet 98547168378947df885c601ccdc85e57
0112 20:51:52.155322 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.204060 (+ 48738us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.250998 (+ 46938us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.259164 (+ 8166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2803095}
I20250112 20:51:52.277889 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 98547168378947df885c601ccdc85e57 (table default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]) successfully deleted
I20250112 20:51:52.278370 1178 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.280230 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a5162a25ce47519a091742f3d5ca31" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:52.280400 774 raft_consensus.cc:3054] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.283720 774 raft_consensus.cc:2463] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:52.284518 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4fa71cf46144629e1fc1f2172fc3cb" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
W20250112 20:51:52.284644 710 consensus_peers.cc:487] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 104d6b4effb84314a6b2b9db883b4366. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:52.285231 710 leader_election.cc:343] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:52.285070 847 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.286166 847 leader_election.cc:304] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:52.286242 907 raft_consensus.cc:3054] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.286325 1163 tablet_bootstrap.cc:492] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.286450 1163 ts_tablet_manager.cc:1397] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.194s user 0.026s sys 0.022s
I20250112 20:51:52.286475 1185 tablet_replica.cc:331] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:52.286650 1185 raft_consensus.cc:2238] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.286604 3305 raft_consensus.cc:2743] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:52.286808 1185 raft_consensus.cc:2267] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.287137 1163 raft_consensus.cc:357] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.287317 1163 raft_consensus.cc:383] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.287339 1185 ts_tablet_manager.cc:1905] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.287381 1163 raft_consensus.cc:738] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.287567 1163 consensus_queue.cc:260] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.290985 907 raft_consensus.cc:2463] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.297605 710 leader_election.cc:304] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:52.298223 1163 ts_tablet_manager.cc:1428] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 20:51:52.298486 1163 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:52.298472 3265 raft_consensus.cc:2798] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.298599 3265 raft_consensus.cc:695] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:52.298841 3265 consensus_queue.cc:237] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.300719 3369 raft_consensus.cc:491] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.300870 3369 raft_consensus.cc:513] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.301463 3369 leader_election.cc:290] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.302295 1186 ts_tablet_manager.cc:1918] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.302438 1186 log.cc:1198] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/65784493c6ef417993b1f6371cab47d6
I20250112 20:51:52.302765 1186 ts_tablet_manager.cc:1939] T 65784493c6ef417993b1f6371cab47d6 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.303323 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17e6d263014448b96c5d1f03013e6e7" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.310185 2447 ts_tablet_manager.cc:1918] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.310894 2447 log.cc:1198] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/c2664a2e98f74f73ab2dc0adaa2335d3
I20250112 20:51:52.311266 2447 ts_tablet_manager.cc:1939] T c2664a2e98f74f73ab2dc0adaa2335d3 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.312448 1178 tablet_bootstrap.cc:654] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.300719 1163 tablet_bootstrap.cc:654] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:52.313445 983 leader_election.cc:343] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: d17e6d263014448b96c5d1f03013e6e7
I20250112 20:51:52.313546 983 leader_election.cc:304] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:52.316606 3369 raft_consensus.cc:2743] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:52.320118 710 consensus_peers.cc:487] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 480d49569886408a8ce781a93bb5d10d. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:52.320605 1164 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.320767 1164 ts_tablet_manager.cc:1397] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.000s
I20250112 20:51:52.321578 1164 raft_consensus.cc:357] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.321825 1164 raft_consensus.cc:383] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.321926 1164 raft_consensus.cc:738] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.322166 1164 consensus_queue.cc:260] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.322305 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.322613 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b00c49f234dc89d63f644febc23fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.323060 1164 ts_tablet_manager.cc:1428] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:52.323421 1164 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.330541 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.333887 1186 tablet_replica.cc:331] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.334070 1186 raft_consensus.cc:2238] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.334218 1186 raft_consensus.cc:2267] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.335346 1186 ts_tablet_manager.cc:1905] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.335573 450 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:52.337172 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 65784493c6ef417993b1f6371cab47d6 (table default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 [id=4130b8a93c4645839e4fad41a2127529]) successfully deleted
I20250112 20:51:52.341962 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 01f6a33e6c95485db80f69fab60d5c9b
I20250112 20:51:52.342880 1186 ts_tablet_manager.cc:1918] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.342981 1186 log.cc:1198] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/24a5162a25ce47519a091742f3d5ca31
I20250112 20:51:52.342970 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 984) took 3494 ms. Trace:
I20250112 20:51:52.343057 2447 rpcz_store.cc:276] 0112 20:51:48.848599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.848653 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:52.271588 (+3422935us) ts_tablet_manager.cc:1145] Deleting tablet c2664a2e98f74f73ab2dc0adaa2335d3
0112 20:51:52.271600 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.302098 (+ 30498us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.310072 (+ 7974us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.342945 (+ 32873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3422720}
I20250112 20:51:52.343248 1186 ts_tablet_manager.cc:1939] T 24a5162a25ce47519a091742f3d5ca31 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.343621 1186 tablet_replica.cc:331] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.343784 1186 raft_consensus.cc:2238] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:52.344000 1186 pending_rounds.cc:62] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:52.344038 1186 pending_rounds.cc:69] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585097172193280 op_type: NO_OP noop_request { }
I20250112 20:51:52.344130 1186 raft_consensus.cc:2883] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:52.344199 1186 raft_consensus.cc:2267] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.346990 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 05e171717b7d462399b141aed9be3c08
I20250112 20:51:52.348937 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.349046 1164 tablet_bootstrap.cc:654] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.349238 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.349558 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f6a33e6c95485db80f69fab60d5c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.349851 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e171717b7d462399b141aed9be3c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.350132 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e171717b7d462399b141aed9be3c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.350637 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e171717b7d462399b141aed9be3c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.350816 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 24a5162a25ce47519a091742f3d5ca31 (table default.Table_7f122343e2d74bf7b3eefea9de3ff725 [id=02dcec3eeef04e78a9c707989a8578bc]) successfully deleted
I20250112 20:51:52.350937 2447 tablet_replica.cc:331] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.351081 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet c2664a2e98f74f73ab2dc0adaa2335d3 (table default.Table_42c5efdd34e643cba03f44dc9c276e53 [id=3ca7b28c5aa44589a0e037a7e4ba20ee]) successfully deleted
I20250112 20:51:52.351246 2447 raft_consensus.cc:2238] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
W20250112 20:51:52.331125 710 consensus_peers.cc:487] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d17e6d263014448b96c5d1f03013e6e7. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:52.339859 742 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.351627 2447 raft_consensus.cc:2267] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:52.351786 3258 log.cc:926] Time spent T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Append to log took a long time: real 0.055s user 0.003s sys 0.000s
I20250112 20:51:52.356251 2447 ts_tablet_manager.cc:1905] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.352417 1047 raft_consensus.cc:3054] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.363601 1185 ts_tablet_manager.cc:1918] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.364888 3373 raft_consensus.cc:491] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.365164 3373 raft_consensus.cc:513] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.365854 1178 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.366047 1178 ts_tablet_manager.cc:1397] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.006s
I20250112 20:51:52.366729 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.366792 1178 raft_consensus.cc:357] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.367000 1178 raft_consensus.cc:383] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.367069 1178 raft_consensus.cc:738] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.367302 1178 consensus_queue.cc:260] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:52.367430 983 leader_election.cc:343] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.367784 1178 ts_tablet_manager.cc:1428] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:52.369518 1186 ts_tablet_manager.cc:1905] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.371376 3305 raft_consensus.cc:491] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.371485 3305 raft_consensus.cc:513] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.372025 3305 leader_election.cc:290] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.372998 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d167065029a46b791d413db8a2c4bcc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:52.373497 847 leader_election.cc:343] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.375000 3373 leader_election.cc:290] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.385115 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 30321942214949b3bcb727d49f4a34d6
I20250112 20:51:52.383893 2447 ts_tablet_manager.cc:1918] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.386018 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 23dc8a695ccd4f88bd108304add01a4e
I20250112 20:51:52.386202 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 30321942214949b3bcb727d49f4a34d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.386508 454 catalog_manager.cc:6596] ReplaceTablet: tablet 23dc8a695ccd4f88bd108304add01a4e deleted and replaced by tablet b071742ab5fc490aa4d53ba4edaa5d63
I20250112 20:51:52.386231 2447 log.cc:1198] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e0fd4177f9394a5a82d4f5e02369b3b6
I20250112 20:51:52.386986 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 30321942214949b3bcb727d49f4a34d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.387362 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dc8a695ccd4f88bd108304add01a4e with delete_type TABLET_DATA_DELETED (replaced by tablet b071742ab5fc490aa4d53ba4edaa5d63) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.387665 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dc8a695ccd4f88bd108304add01a4e with delete_type TABLET_DATA_DELETED (replaced by tablet b071742ab5fc490aa4d53ba4edaa5d63) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.392951 2447 ts_tablet_manager.cc:1939] T e0fd4177f9394a5a82d4f5e02369b3b6 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.393004 1186 ts_tablet_manager.cc:1918] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.393108 1186 log.cc:1198] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/615d75bc22df45cc876788a266e983cc
I20250112 20:51:52.393443 1186 ts_tablet_manager.cc:1939] T 615d75bc22df45cc876788a266e983cc P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.394392 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 985) took 3545 ms. Trace:
I20250112 20:51:52.395079 2447 rpcz_store.cc:276] 0112 20:51:48.848720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.848771 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:52.350890 (+3502119us) ts_tablet_manager.cc:1145] Deleting tablet e0fd4177f9394a5a82d4f5e02369b3b6
0112 20:51:52.350900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.366518 (+ 15618us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.383799 (+ 17281us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.394374 (+ 10575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3501909}
I20250112 20:51:52.395818 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e0fd4177f9394a5a82d4f5e02369b3b6 (table default.Table_cba467c888b84da6964f049247721310 [id=c0f4f9a776794374b5cfc063becc2876]) successfully deleted
I20250112 20:51:52.396268 2447 tablet_replica.cc:331] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.397333 454 catalog_manager.cc:5526] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:52.397745 454 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:52.398039 2447 raft_consensus.cc:2238] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:52.399055 2447 raft_consensus.cc:2267] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.399904 2447 ts_tablet_manager.cc:1905] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.400339 889 tablet_service.cc:1467] Processing CreateTablet for tablet 748253309ff2486fbaaa3217076556f4 (DEFAULT_TABLE table=default.Table_7209914d98d7435883751f90fcdce586 [id=bd5f33d356054d58bd794f9bae0a8b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.401489 449 master_service.cc:946] ReplaceTablet: received request to replace tablet aeac9a4034d949e3bbec76b9c32642cf from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:52.401671 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 66fa87d8f0c64f7aa616882fe399ef0e
I20250112 20:51:52.403499 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.402850 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.404636 447 master_service.cc:946] ReplaceTablet: received request to replace tablet e3648997eb1241edaf56c825bf76b240 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:52.406702 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 748253309ff2486fbaaa3217076556f4 (DEFAULT_TABLE table=default.Table_7209914d98d7435883751f90fcdce586 [id=bd5f33d356054d58bd794f9bae0a8b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.407087 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748253309ff2486fbaaa3217076556f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.411866 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet fa4e7313e9934f50816aba955976311f
I20250112 20:51:52.412425 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4e7313e9934f50816aba955976311f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.413030 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet aeac9a4034d949e3bbec76b9c32642cf
I20250112 20:51:52.413535 449 catalog_manager.cc:6596] ReplaceTablet: tablet aeac9a4034d949e3bbec76b9c32642cf deleted and replaced by tablet 8582f6d6b1204d42b10e3c0e7cea489a
I20250112 20:51:52.413735 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:52.413800 891 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4e7313e9934f50816aba955976311f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.414188 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet aeac9a4034d949e3bbec76b9c32642cf with delete_type TABLET_DATA_DELETED (replaced by tablet 8582f6d6b1204d42b10e3c0e7cea489a) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.414530 891 tablet_service.cc:1514] Processing DeleteTablet for tablet aeac9a4034d949e3bbec76b9c32642cf with delete_type TABLET_DATA_DELETED (replaced by tablet 8582f6d6b1204d42b10e3c0e7cea489a) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.415884 1163 tablet_bootstrap.cc:492] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.416074 1163 ts_tablet_manager.cc:1397] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.010s
I20250112 20:51:52.418033 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26d40d5bd44e7385e805060a9ab2fc (DEFAULT_TABLE table=default.Table_89746631f84f4bb097908ead01455c16 [id=79cb10ea6de147f4a5210b2d97387d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.419190 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26d40d5bd44e7385e805060a9ab2fc (DEFAULT_TABLE table=default.Table_89746631f84f4bb097908ead01455c16 [id=79cb10ea6de147f4a5210b2d97387d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.419556 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26d40d5bd44e7385e805060a9ab2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.421641 884 tablet_service.cc:1467] Processing CreateTablet for tablet 47248e998d104da5b29e139885195c3b (DEFAULT_TABLE table=default.Table_b95e22f892d24fe390619250ef5fba8d [id=5f265788b53b4d6092013a9f7ccd311d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.421996 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.422035 879 tablet_service.cc:1514] Processing DeleteTablet for tablet bf66492d01a64eb1b842acc279b8ee09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:43 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.423154 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:52.423568 1163 raft_consensus.cc:357] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.423633 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 47248e998d104da5b29e139885195c3b (DEFAULT_TABLE table=default.Table_b95e22f892d24fe390619250ef5fba8d [id=5f265788b53b4d6092013a9f7ccd311d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.423739 1163 raft_consensus.cc:383] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.423789 1163 raft_consensus.cc:738] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.423959 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47248e998d104da5b29e139885195c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.424017 1163 consensus_queue.cc:260] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.424494 1163 ts_tablet_manager.cc:1428] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250112 20:51:52.425014 1163 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:52.426301 879 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6a65a2ae504eaf8c66e705b46b00b4 (DEFAULT_TABLE table=default.Table_01d0b758a621413cb1fb3e51347fde91 [id=3025651d9b124550a88238198ca6f200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.426837 1163 tablet_bootstrap.cc:654] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.427462 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6a65a2ae504eaf8c66e705b46b00b4 (DEFAULT_TABLE table=default.Table_01d0b758a621413cb1fb3e51347fde91 [id=3025651d9b124550a88238198ca6f200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.427783 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6a65a2ae504eaf8c66e705b46b00b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.431084 892 tablet_service.cc:1467] Processing CreateTablet for tablet f19d886a7e584bbdb38b135bb8654edb (DEFAULT_TABLE table=default.Table_44d406e2cfe24b0db868598432e2a504 [id=2d8a4cfb464e4819985350715d982fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.432582 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f19d886a7e584bbdb38b135bb8654edb (DEFAULT_TABLE table=default.Table_44d406e2cfe24b0db868598432e2a504 [id=2d8a4cfb464e4819985350715d982fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.432907 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19d886a7e584bbdb38b135bb8654edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.434738 885 tablet_service.cc:1467] Processing CreateTablet for tablet c9206fc7b077452eb90acb15a50e71a4 (DEFAULT_TABLE table=default.Table_fdc1f15839ae4f91a0b63261f416283a [id=cc2fb3604e634e7d9772a73fc1672037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.435252 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9206fc7b077452eb90acb15a50e71a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.439929 1014 tablet_service.cc:1467] Processing CreateTablet for tablet c9206fc7b077452eb90acb15a50e71a4 (DEFAULT_TABLE table=default.Table_fdc1f15839ae4f91a0b63261f416283a [id=cc2fb3604e634e7d9772a73fc1672037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.440078 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet db9bab2477a94c169817d9de8aaad1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:48 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.440260 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9206fc7b077452eb90acb15a50e71a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.444458 885 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8402b859417384702ed7021fe1fc (DEFAULT_TABLE table=default.Table_b3ec982cbf00412692d213f224f81413 [id=add51fabe4da44e19149588a01ab64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.445012 1023 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8402b859417384702ed7021fe1fc (DEFAULT_TABLE table=default.Table_b3ec982cbf00412692d213f224f81413 [id=add51fabe4da44e19149588a01ab64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.445338 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcb8402b859417384702ed7021fe1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.446295 882 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2f3e0b9f05455090bddaa15fe0a280 (DEFAULT_TABLE table=default.Table_8d9f63102f7546f9b40db699c7d49be9 [id=41349c47e2a5449ba1fd4fcebb8a2d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.447242 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2f3e0b9f05455090bddaa15fe0a280 (DEFAULT_TABLE table=default.Table_8d9f63102f7546f9b40db699c7d49be9 [id=41349c47e2a5449ba1fd4fcebb8a2d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.447749 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcb8402b859417384702ed7021fe1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.448091 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2f3e0b9f05455090bddaa15fe0a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.451887 888 tablet_service.cc:1467] Processing CreateTablet for tablet c2881ca71d3949ee82b476c765387a69 (DEFAULT_TABLE table=default.Table_d019aad7635645ecbede54659d8e4f91 [id=145f77d612764ba2aa000260a2326051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.452201 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2881ca71d3949ee82b476c765387a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.457154 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a72551b18fc4484c89e651cf3af7ff94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.457705 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72551b18fc4484c89e651cf3af7ff94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.457182 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4be3e81d23584e39ac24e2ed93876f47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.459013 2447 ts_tablet_manager.cc:1918] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.459136 2447 log.cc:1198] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f875f075e98e45499291b519caf03184
W20250112 20:51:52.459184 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4be3e81d23584e39ac24e2ed93876f47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.459460 2447 ts_tablet_manager.cc:1939] T f875f075e98e45499291b519caf03184 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.460517 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_251238528b1d4607bb3bf46da6d40a56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.461077 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_251238528b1d4607bb3bf46da6d40a56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.465317 1022 tablet_service.cc:1467] Processing CreateTablet for tablet c2881ca71d3949ee82b476c765387a69 (DEFAULT_TABLE table=default.Table_d019aad7635645ecbede54659d8e4f91 [id=145f77d612764ba2aa000260a2326051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.467797 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ccc15f3eb2cd41aeac09ecb6e04fa79f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.468169 1178 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.468072 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e929ae2e87394e6fbec49a339af2de71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.468389 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc15f3eb2cd41aeac09ecb6e04fa79f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:52.468590 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e929ae2e87394e6fbec49a339af2de71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.468678 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fe4fe28686e24c5c8505eeda33fe9440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.469017 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_ab8ef713e8a7432fa513234210a4cb6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.469202 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe4fe28686e24c5c8505eeda33fe9440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:52.469522 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab8ef713e8a7432fa513234210a4cb6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.473959 1186 tablet_replica.cc:331] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.473945 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 615d75bc22df45cc876788a266e983cc (table default.Table_b5d6bc8b522a4be197bd045c77d4a5f4 [id=3066f1ca47fe4a39841cfd5b6028f16c]) successfully deleted
I20250112 20:51:52.474192 1186 raft_consensus.cc:2238] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.474344 1186 raft_consensus.cc:2267] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.474725 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748253309ff2486fbaaa3217076556f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.474857 1186 ts_tablet_manager.cc:1905] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.474912 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47248e998d104da5b29e139885195c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.477521 888 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa531994c84d1eb183b7805d48a63d (DEFAULT_TABLE table=default.Table_9db0cdd2cd10494c9bff7b59017806a0 [id=d77a1f14c19c434eb9cc3796e94cf79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.478371 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fa531994c84d1eb183b7805d48a63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.480631 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6a65a2ae504eaf8c66e705b46b00b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.480664 1186 ts_tablet_manager.cc:1918] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.480749 1186 log.cc:1198] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/db9bab2477a94c169817d9de8aaad1c0
I20250112 20:51:52.481047 1186 ts_tablet_manager.cc:1939] T db9bab2477a94c169817d9de8aaad1c0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.481376 1186 tablet_replica.cc:331] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.481554 1186 raft_consensus.cc:2238] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.481696 1186 raft_consensus.cc:2267] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.481690 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet db9bab2477a94c169817d9de8aaad1c0 (table default.Table_c0ad7aebc1e245b2bcd5c4af8c8211c6 [id=2f68c758f6554102961912265cf48921]) successfully deleted
I20250112 20:51:52.482270 1186 ts_tablet_manager.cc:1905] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.483165 1178 tablet_bootstrap.cc:654] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.484624 1163 tablet_bootstrap.cc:492] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.485008 1163 ts_tablet_manager.cc:1397] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.001s
I20250112 20:51:52.485792 1163 raft_consensus.cc:357] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.486093 1163 raft_consensus.cc:383] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.486197 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa531994c84d1eb183b7805d48a63d (DEFAULT_TABLE table=default.Table_9db0cdd2cd10494c9bff7b59017806a0 [id=d77a1f14c19c434eb9cc3796e94cf79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.485006 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19d886a7e584bbdb38b135bb8654edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.491091 889 tablet_service.cc:1467] Processing CreateTablet for tablet 45f7bdf88ab942c289869ceba4b72bfd (DEFAULT_TABLE table=default.Table_6a80d9a8edfd459aad1ba74017ec03ce [id=a1c837e6dc3240928dc2e51577963009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.491537 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7bdf88ab942c289869ceba4b72bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.493355 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26d40d5bd44e7385e805060a9ab2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.497638 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9791295b4d4b49d796e5c6552f72adbd (DEFAULT_TABLE table=default.Table_f98e8647f21b468c96340e5662c7d5f3 [id=f4304a292a264d0d822aa470f61cfd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.497967 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9791295b4d4b49d796e5c6552f72adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.500391 1186 ts_tablet_manager.cc:1918] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.500618 1186 log.cc:1198] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5d167065029a46b791d413db8a2c4bcc
I20250112 20:51:52.500998 1186 ts_tablet_manager.cc:1939] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.502764 1186 tablet_replica.cc:331] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.502995 1186 raft_consensus.cc:2238] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.503135 1186 raft_consensus.cc:2267] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.486207 1163 raft_consensus.cc:738] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.503679 1186 ts_tablet_manager.cc:1905] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.503741 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5d167065029a46b791d413db8a2c4bcc (table default.Table_6e73aeaec7af4cd0952e8f427103f61f [id=0f5678b6533e42d190d24c6958091caa]) successfully deleted
I20250112 20:51:52.503674 1163 consensus_queue.cc:260] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:52.504122 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 008d69d199d64bdf85da17d8d3bf6264 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 008d69d199d64bdf85da17d8d3bf6264 already in progress: creating tablet
W20250112 20:51:52.504333 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 417ada5111454c54a85cc95db5974bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417ada5111454c54a85cc95db5974bf9 already in progress: creating tablet
I20250112 20:51:52.504482 1163 ts_tablet_manager.cc:1428] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250112 20:51:52.507709 891 tablet_service.cc:1467] Processing CreateTablet for tablet e2fe642ca7f64d9fa5c351a4854bd044 (DEFAULT_TABLE table=default.Table_6da9b6b44a4440dd8378e22ca2f174ee [id=4b6ab842eb3d4111ad7f6a61eb37459a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.508081 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2fe642ca7f64d9fa5c351a4854bd044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.509542 1186 ts_tablet_manager.cc:1918] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.509639 1186 log.cc:1198] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/725e56fa8ac343e5b1ec1214da02b7d4
I20250112 20:51:52.509940 1186 ts_tablet_manager.cc:1939] T 725e56fa8ac343e5b1ec1214da02b7d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.510473 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 725e56fa8ac343e5b1ec1214da02b7d4 (table default.Table_be8270d1ca354e748dac253a4aeb09a3 [id=a63f77f9d86346d0aed333ccca1a28ce]) successfully deleted
W20250112 20:51:52.511296 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 782a195e68124cd391d0394bcb5ec8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 782a195e68124cd391d0394bcb5ec8b2 already in progress: creating tablet
W20250112 20:51:52.511525 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet df292cbdf58b4fcd889fcb63335b37d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df292cbdf58b4fcd889fcb63335b37d9 already in progress: creating tablet
I20250112 20:51:52.511512 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 45f7bdf88ab942c289869ceba4b72bfd (DEFAULT_TABLE table=default.Table_6a80d9a8edfd459aad1ba74017ec03ce [id=a1c837e6dc3240928dc2e51577963009]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:52.511705 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 91243c9c6deb447e8246f7abe2691f55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91243c9c6deb447e8246f7abe2691f55 already in progress: creating tablet
W20250112 20:51:52.511909 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 01f6a33e6c95485db80f69fab60d5c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01f6a33e6c95485db80f69fab60d5c9b already in progress: creating tablet
I20250112 20:51:52.511897 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 9791295b4d4b49d796e5c6552f72adbd (DEFAULT_TABLE table=default.Table_f98e8647f21b468c96340e5662c7d5f3 [id=f4304a292a264d0d822aa470f61cfd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.512539 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e2fe642ca7f64d9fa5c351a4854bd044 (DEFAULT_TABLE table=default.Table_6da9b6b44a4440dd8378e22ca2f174ee [id=4b6ab842eb3d4111ad7f6a61eb37459a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:52.513283 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 05e171717b7d462399b141aed9be3c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05e171717b7d462399b141aed9be3c08 already in progress: creating tablet
I20250112 20:51:52.516098 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2881ca71d3949ee82b476c765387a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.519112 888 tablet_service.cc:1467] Processing CreateTablet for tablet d8ea5b6d791d4891a1bb008879eba941 (DEFAULT_TABLE table=default.Table_84f9735e457340a490e02f9c6983d2e5 [id=b7e954dbaf8347be89a9786b23ab44d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:52.520949 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 30321942214949b3bcb727d49f4a34d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30321942214949b3bcb727d49f4a34d6 already in progress: creating tablet
I20250112 20:51:52.363812 3114 consensus_queue.cc:1035] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.157s
I20250112 20:51:52.366380 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.521318 774 raft_consensus.cc:2463] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
W20250112 20:51:52.521481 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 23dc8a695ccd4f88bd108304add01a4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23dc8a695ccd4f88bd108304add01a4e already in progress: creating tablet
W20250112 20:51:52.521764 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 66fa87d8f0c64f7aa616882fe399ef0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66fa87d8f0c64f7aa616882fe399ef0e already in progress: creating tablet
I20250112 20:51:52.521910 983 leader_election.cc:304] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:52.521984 1186 tablet_replica.cc:331] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.522176 1186 raft_consensus.cc:2238] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.522325 1186 raft_consensus.cc:2267] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.522369 3373 raft_consensus.cc:2798] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.522431 3373 raft_consensus.cc:491] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.522488 3373 raft_consensus.cc:3054] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.372869 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d167065029a46b791d413db8a2c4bcc" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:52.522754 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet fa4e7313e9934f50816aba955976311f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa4e7313e9934f50816aba955976311f already in progress: creating tablet
W20250112 20:51:52.522992 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet aeac9a4034d949e3bbec76b9c32642cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeac9a4034d949e3bbec76b9c32642cf already in progress: creating tablet
I20250112 20:51:52.385746 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 30321942214949b3bcb727d49f4a34d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.386598 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 23dc8a695ccd4f88bd108304add01a4e with delete_type TABLET_DATA_DELETED (replaced by tablet b071742ab5fc490aa4d53ba4edaa5d63) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.403144 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fa87d8f0c64f7aa616882fe399ef0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.405746 1164 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.413282 756 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4e7313e9934f50816aba955976311f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.523468 1164 ts_tablet_manager.cc:1397] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.200s user 0.009s sys 0.000s
I20250112 20:51:52.524226 1164 raft_consensus.cc:357] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.524392 1164 raft_consensus.cc:383] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.524446 1164 raft_consensus.cc:738] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.524657 1186 ts_tablet_manager.cc:1905] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.414734 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.414780 741 tablet_service.cc:1514] Processing DeleteTablet for tablet aeac9a4034d949e3bbec76b9c32642cf with delete_type TABLET_DATA_DELETED (replaced by tablet 8582f6d6b1204d42b10e3c0e7cea489a) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.525007 3373 raft_consensus.cc:513] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.420230 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1a26d40d5bd44e7385e805060a9ab2fc (DEFAULT_TABLE table=default.Table_89746631f84f4bb097908ead01455c16 [id=79cb10ea6de147f4a5210b2d97387d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.525393 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a26d40d5bd44e7385e805060a9ab2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.525579 3373 leader_election.cc:290] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.363719 1185 log.cc:1198] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/3dba3d9315db4b9292288ee62c2244bc
I20250112 20:51:52.526237 1185 ts_tablet_manager.cc:1939] T 3dba3d9315db4b9292288ee62c2244bc P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.526507 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1052) took 3174 ms. Trace:
I20250112 20:51:52.526566 1185 rpcz_store.cc:276] 0112 20:51:49.352434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.352701 (+ 267us) service_pool.cc:225] Handling call
0112 20:51:52.286426 (+2933725us) ts_tablet_manager.cc:1145] Deleting tablet 3dba3d9315db4b9292288ee62c2244bc
0112 20:51:52.286435 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.353281 (+ 66846us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.363514 (+ 10233us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.526495 (+162981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2933523}
I20250112 20:51:52.526916 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 3dba3d9315db4b9292288ee62c2244bc (table default.Table_5b2f736731d74cdfb099e7a498bf92a0 [id=76b37e8b71714bdc8a90e08e06a03a02]) successfully deleted
I20250112 20:51:52.425137 752 tablet_service.cc:1467] Processing CreateTablet for tablet 47248e998d104da5b29e139885195c3b (DEFAULT_TABLE table=default.Table_b95e22f892d24fe390619250ef5fba8d [id=5f265788b53b4d6092013a9f7ccd311d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.529191 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47248e998d104da5b29e139885195c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.428700 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6a65a2ae504eaf8c66e705b46b00b4 (DEFAULT_TABLE table=default.Table_01d0b758a621413cb1fb3e51347fde91 [id=3025651d9b124550a88238198ca6f200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.532999 755 tablet_service.cc:1467] Processing CreateTablet for tablet 11be9d5cff2043df81ecfc6255c40004 (DEFAULT_TABLE table=default.Table_91202e68552445aeab8e24f4978d8203 [id=95b98b692428460785ee9378132e4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.533095 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6a65a2ae504eaf8c66e705b46b00b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.533257 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11be9d5cff2043df81ecfc6255c40004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.533537 741 tablet_service.cc:1467] Processing CreateTablet for tablet d8ea5b6d791d4891a1bb008879eba941 (DEFAULT_TABLE table=default.Table_84f9735e457340a490e02f9c6983d2e5 [id=b7e954dbaf8347be89a9786b23ab44d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.533788 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ea5b6d791d4891a1bb008879eba941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.433697 739 tablet_service.cc:1467] Processing CreateTablet for tablet f19d886a7e584bbdb38b135bb8654edb (DEFAULT_TABLE table=default.Table_44d406e2cfe24b0db868598432e2a504 [id=2d8a4cfb464e4819985350715d982fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.441395 748 tablet_service.cc:1467] Processing CreateTablet for tablet c9206fc7b077452eb90acb15a50e71a4 (DEFAULT_TABLE table=default.Table_fdc1f15839ae4f91a0b63261f416283a [id=cc2fb3604e634e7d9772a73fc1672037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.444243 749 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8402b859417384702ed7021fe1fc (DEFAULT_TABLE table=default.Table_b3ec982cbf00412692d213f224f81413 [id=add51fabe4da44e19149588a01ab64af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.449023 737 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2f3e0b9f05455090bddaa15fe0a280 (DEFAULT_TABLE table=default.Table_8d9f63102f7546f9b40db699c7d49be9 [id=41349c47e2a5449ba1fd4fcebb8a2d0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.488679 738 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa531994c84d1eb183b7805d48a63d (DEFAULT_TABLE table=default.Table_9db0cdd2cd10494c9bff7b59017806a0 [id=d77a1f14c19c434eb9cc3796e94cf79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.495054 745 tablet_service.cc:1467] Processing CreateTablet for tablet 45f7bdf88ab942c289869ceba4b72bfd (DEFAULT_TABLE table=default.Table_6a80d9a8edfd459aad1ba74017ec03ce [id=a1c837e6dc3240928dc2e51577963009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.501860 740 tablet_service.cc:1467] Processing CreateTablet for tablet 9791295b4d4b49d796e5c6552f72adbd (DEFAULT_TABLE table=default.Table_f98e8647f21b468c96340e5662c7d5f3 [id=f4304a292a264d0d822aa470f61cfd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.515592 747 tablet_service.cc:1467] Processing CreateTablet for tablet e2fe642ca7f64d9fa5c351a4854bd044 (DEFAULT_TABLE table=default.Table_6da9b6b44a4440dd8378e22ca2f174ee [id=4b6ab842eb3d4111ad7f6a61eb37459a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.407364 754 tablet_service.cc:1467] Processing CreateTablet for tablet 748253309ff2486fbaaa3217076556f4 (DEFAULT_TABLE table=default.Table_7209914d98d7435883751f90fcdce586 [id=bd5f33d356054d58bd794f9bae0a8b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.438928 3265 raft_consensus.cc:491] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.538595 3265 raft_consensus.cc:513] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.539196 3265 leader_election.cc:290] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.539427 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:52.477064 746 tablet_service.cc:1467] Processing CreateTablet for tablet c2881ca71d3949ee82b476c765387a69 (DEFAULT_TABLE table=default.Table_d019aad7635645ecbede54659d8e4f91 [id=145f77d612764ba2aa000260a2326051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.501935 3121 raft_consensus.cc:491] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.539991 3121 raft_consensus.cc:513] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.540441 3121 leader_election.cc:290] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:52.541042 983 leader_election.cc:343] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.542430 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cf6ee0c97faa43b0ab60b144c37f65e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.542976 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf6ee0c97faa43b0ab60b144c37f65e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.544227 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f020d2e2fdf24b08ba8be0a7cfb00340" } new_table_name: "default.Table_26eeb9cb2a0249928bf1dbc292e71d20" modify_external_catalogs: true
I20250112 20:51:52.544205 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3c90a3a974a141558db73a01cfa97722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.544634 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c90a3a974a141558db73a01cfa97722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.545449 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5c0a48d4b1f44f1e9d75d400ebab7e4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.545550 847 leader_election.cc:343] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.545655 847 leader_election.cc:304] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:52.545801 847 raft_consensus.cc:2681] T 5d167065029a46b791d413db8a2c4bcc P 87ae45957c64469d96dc92d606b5b4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 20:51:52.545871 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c0a48d4b1f44f1e9d75d400ebab7e4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.546420 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf66492d01a64eb1b842acc279b8ee09" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.546612 908 raft_consensus.cc:2463] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
W20250112 20:51:52.546727 710 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 65784493c6ef417993b1f6371cab47d6
I20250112 20:51:52.546869 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dafaa3a1a3d4d8bb8aa0448a61e237c" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:52.547026 758 raft_consensus.cc:3054] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.547168 710 leader_election.cc:304] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.547621 3265 raft_consensus.cc:2798] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.547688 3265 raft_consensus.cc:491] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.547741 3265 raft_consensus.cc:3054] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.548369 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2050ea3fd15848d7b59a8b8020368824" } new_table_name: "default.Table_04fc014a5d4a4a29952e098874392d12" modify_external_catalogs: true
I20250112 20:51:52.549607 758 raft_consensus.cc:2463] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:52.550173 3265 raft_consensus.cc:513] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.550288 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_a06a212d24b846debb4ec624d3a1f24d" } new_table_name: "default.Table_a4126d4b12a545df8133eaf331d701ee" modify_external_catalogs: true
I20250112 20:51:52.550570 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 11be9d5cff2043df81ecfc6255c40004 (DEFAULT_TABLE table=default.Table_91202e68552445aeab8e24f4978d8203 [id=95b98b692428460785ee9378132e4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.550777 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf66492d01a64eb1b842acc279b8ee09" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:52.550989 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11be9d5cff2043df81ecfc6255c40004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.550997 1039 raft_consensus.cc:2463] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:52.552238 3265 leader_election.cc:290] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.553607 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf66492d01a64eb1b842acc279b8ee09" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:52.553766 907 raft_consensus.cc:3054] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.553794 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65784493c6ef417993b1f6371cab47d6" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:52.556270 907 raft_consensus.cc:2463] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.556226 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_875e19c996d5437894e99baaa6fcffbf" } new_table_name: "default.Table_6c5f1c8deed84291ae0f2a2c8889ad88" modify_external_catalogs: true
I20250112 20:51:52.524650 1164 consensus_queue.cc:260] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.557953 1164 ts_tablet_manager.cc:1428] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.034s user 0.001s sys 0.000s
I20250112 20:51:52.558766 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf66492d01a64eb1b842acc279b8ee09" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:52.558946 1044 raft_consensus.cc:3054] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.561429 1044 raft_consensus.cc:2463] T bf66492d01a64eb1b842acc279b8ee09 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.561445 983 leader_election.cc:304] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:52.561926 710 leader_election.cc:304] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:52.562352 710 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.562451 710 leader_election.cc:304] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:52.562629 3121 raft_consensus.cc:2798] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.562623 3373 raft_consensus.cc:2798] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.562714 3121 raft_consensus.cc:695] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:52.562862 3373 raft_consensus.cc:695] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:52.562917 3121 consensus_queue.cc:237] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.563112 3373 consensus_queue.cc:237] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.564070 3391 raft_consensus.cc:2743] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:52.566671 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85683e97dae44399b1dc7bbd54d62eb6 consensus state has no leader
I20250112 20:51:52.567293 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 986) took 3718 ms. Trace:
I20250112 20:51:52.567579 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fa531994c84d1eb183b7805d48a63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.567464 2447 rpcz_store.cc:276] 0112 20:51:48.848869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.848935 (+ 66us) service_pool.cc:225] Handling call
0112 20:51:52.396211 (+3547276us) ts_tablet_manager.cc:1145] Deleting tablet f875f075e98e45499291b519caf03184
0112 20:51:52.396224 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.408541 (+ 12317us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.458931 (+ 50390us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.567273 (+108342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3547147}
I20250112 20:51:52.568090 2447 tablet_replica.cc:331] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.568346 2447 raft_consensus.cc:2238] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.568542 2447 raft_consensus.cc:2267] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.569221 2447 ts_tablet_manager.cc:1905] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.571519 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2fe642ca7f64d9fa5c351a4854bd044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.571841 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f875f075e98e45499291b519caf03184 (table default.Table_0a39583e302f435bb0ae3f12b56e8e51 [id=a41262782d3c41bea2fc1decd8ba5a95]) successfully deleted
I20250112 20:51:52.572944 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d8ea5b6d791d4891a1bb008879eba941 (DEFAULT_TABLE table=default.Table_84f9735e457340a490e02f9c6983d2e5 [id=b7e954dbaf8347be89a9786b23ab44d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.573415 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ea5b6d791d4891a1bb008879eba941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.574035 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9791295b4d4b49d796e5c6552f72adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.576108 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2f3e0b9f05455090bddaa15fe0a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.576617 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ea5b6d791d4891a1bb008879eba941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.577018 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7bdf88ab942c289869ceba4b72bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.579262 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3591211d8ed5415b820beae2848ac1f5" } modify_external_catalogs: true
I20250112 20:51:52.579463 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3591211d8ed5415b820beae2848ac1f5" } modify_external_catalogs: true
I20250112 20:51:52.580327 1186 ts_tablet_manager.cc:1918] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.580422 1186 log.cc:1198] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/ab9be1f065ec4605a0a3fe69770793d0
I20250112 20:51:52.580791 1186 ts_tablet_manager.cc:1939] T ab9be1f065ec4605a0a3fe69770793d0 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.580757 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_1f79972c8942460f988e2d87f374fc5e" } new_table_name: "default.Table_ad29dfc257d84f3fb66f45aa86e14ece" modify_external_catalogs: true
I20250112 20:51:52.581066 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_16215b3f2b9d4bba913676a2f2590409" } modify_external_catalogs: true
I20250112 20:51:52.582201 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_16215b3f2b9d4bba913676a2f2590409" } modify_external_catalogs: true
I20250112 20:51:52.583725 879 tablet_service.cc:1467] Processing CreateTablet for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb (DEFAULT_TABLE table=default.Table_93db35038f744bfdbd5526c04bc025a4 [id=e6e813707db34939a81d8c37721cfccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.587749 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb (DEFAULT_TABLE table=default.Table_93db35038f744bfdbd5526c04bc025a4 [id=e6e813707db34939a81d8c37721cfccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.588128 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.591755 756 tablet_service.cc:1467] Processing CreateTablet for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb (DEFAULT_TABLE table=default.Table_93db35038f744bfdbd5526c04bc025a4 [id=e6e813707db34939a81d8c37721cfccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.592154 1163 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:52.596830 1163 tablet_bootstrap.cc:654] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.598089 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19d886a7e584bbdb38b135bb8654edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.599874 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1051) took 3247 ms. Trace:
I20250112 20:51:52.599977 1185 rpcz_store.cc:276] 0112 20:51:49.352196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.352922 (+ 726us) service_pool.cc:225] Handling call
0112 20:51:52.599757 (+3246835us) ts_tablet_manager.cc:1145] Deleting tablet cbc22207e9204b73877c74f61ebba8a8
0112 20:51:52.599767 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.599856 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3246678}
I20250112 20:51:52.600720 3284 raft_consensus.cc:491] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.600844 3284 raft_consensus.cc:513] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.601089 751 tablet_service.cc:1467] Processing CreateTablet for tablet 86b51849dd2647508ec6ec06503ff850 (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.601495 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b51849dd2647508ec6ec06503ff850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.601615 3284 leader_election.cc:290] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.602701 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725e56fa8ac343e5b1ec1214da02b7d4" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.602869 758 raft_consensus.cc:2463] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:52.603042 1178 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.603164 1178 ts_tablet_manager.cc:1397] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.135s user 0.002s sys 0.002s
I20250112 20:51:52.603248 983 leader_election.cc:304] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.604199 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9206fc7b077452eb90acb15a50e71a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.604768 3373 raft_consensus.cc:2798] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.604864 3373 raft_consensus.cc:491] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.604975 3373 raft_consensus.cc:3054] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.607314 3373 raft_consensus.cc:513] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.608166 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725e56fa8ac343e5b1ec1214da02b7d4" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:52.608310 758 raft_consensus.cc:3054] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.609006 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcb8402b859417384702ed7021fe1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.610757 758 raft_consensus.cc:2463] T 725e56fa8ac343e5b1ec1214da02b7d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:52.611155 983 leader_election.cc:304] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.611393 3284 raft_consensus.cc:2798] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.611399 882 tablet_service.cc:1467] Processing CreateTablet for tablet 11be9d5cff2043df81ecfc6255c40004 (DEFAULT_TABLE table=default.Table_91202e68552445aeab8e24f4978d8203 [id=95b98b692428460785ee9378132e4009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.611482 3284 raft_consensus.cc:695] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:52.611696 3284 consensus_queue.cc:237] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.613730 1178 raft_consensus.cc:357] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.613903 1178 raft_consensus.cc:383] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.613945 1178 raft_consensus.cc:738] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.614147 1178 consensus_queue.cc:260] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.614629 1178 ts_tablet_manager.cc:1428] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 20:51:52.631955 3373 leader_election.cc:290] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.636096 3121 raft_consensus.cc:491] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.636307 3121 raft_consensus.cc:513] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.637683 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:52.638222 710 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.638398 3121 leader_election.cc:290] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.645311 884 tablet_service.cc:1467] Processing CreateTablet for tablet 86b51849dd2647508ec6ec06503ff850 (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:52.646353 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet cbc22207e9204b73877c74f61ebba8a8 because the tablet was not found. No further retry: Not found: Tablet not found: cbc22207e9204b73877c74f61ebba8a8
I20250112 20:51:52.648001 2447 ts_tablet_manager.cc:1918] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.649005 1185 tablet_replica.cc:331] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:52.649227 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fa531994c84d1eb183b7805d48a63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.649364 1185 raft_consensus.cc:2238] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:52.649892 1185 raft_consensus.cc:2267] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.650516 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e5949130bc9c488da66e7ebeeaf6cc8c" } modify_external_catalogs: true
I20250112 20:51:52.650889 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11813bf0d59d467d8f57da28e92d1b9f" } modify_external_catalogs: true
I20250112 20:51:52.654326 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7bdf88ab942c289869ceba4b72bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.654999 1185 ts_tablet_manager.cc:1905] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.663095 3305 consensus_queue.cc:1035] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:52.664633 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:52.667285 2447 log.cc:1198] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9989f3dc61df4f75b9bdb85ba3ca3f37
I20250112 20:51:52.668558 3304 consensus_queue.cc:1035] T d72ae28879dc45429de2ab0d66276255 P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:52.669389 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_04069d4b7bad4757988c3e7e8ab02b44" } modify_external_catalogs: true
W20250112 20:51:52.673074 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c07855f83f484e62b400775f71925857 consensus state has no leader
I20250112 20:51:52.673182 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e5949130bc9c488da66e7ebeeaf6cc8c" } modify_external_catalogs: true
I20250112 20:51:52.674719 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_3d064f1656bd4dfc83bf03f62daea2d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.675217 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d064f1656bd4dfc83bf03f62daea2d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.675371 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_11813bf0d59d467d8f57da28e92d1b9f" } modify_external_catalogs: true
W20250112 20:51:52.673094 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67e7d4e9061b4ca4aa7f2a158ea54a7d consensus state has no leader
W20250112 20:51:52.676426 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3cfbcab367040b88dd76ac37e0791c6 consensus state has no leader
I20250112 20:51:52.681797 750 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.682014 3265 raft_consensus.cc:491] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.682155 3265 raft_consensus.cc:513] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.682719 3265 leader_election.cc:290] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.657660 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 86b51849dd2647508ec6ec06503ff850 (DEFAULT_TABLE table=default.Table_77867f973c5547568b64d2b3448da1d4 [id=ac970ed0c7f84cfd983982a4f9cf0621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.683477 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9bab2477a94c169817d9de8aaad1c0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:52.688197 1047 raft_consensus.cc:2463] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:52.688392 2447 ts_tablet_manager.cc:1939] T 9989f3dc61df4f75b9bdb85ba3ca3f37 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.688400 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b51849dd2647508ec6ec06503ff850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.689180 710 leader_election.cc:304] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.689438 3265 raft_consensus.cc:2798] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.689509 3265 raft_consensus.cc:491] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.689625 3265 raft_consensus.cc:3054] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.691931 3265 raft_consensus.cc:513] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.692480 3265 leader_election.cc:290] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:52.693181 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9bab2477a94c169817d9de8aaad1c0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:52.693397 1047 raft_consensus.cc:3054] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.695925 1047 raft_consensus.cc:2463] T db9bab2477a94c169817d9de8aaad1c0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.696218 3130 raft_consensus.cc:491] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.696317 3130 raft_consensus.cc:513] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.696854 3130 leader_election.cc:290] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.697237 710 leader_election.cc:304] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.697623 3265 raft_consensus.cc:2798] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.697721 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b00c49f234dc89d63f644febc23fb" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:52.698014 774 raft_consensus.cc:2388] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:52.697741 3265 raft_consensus.cc:695] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:52.698539 3265 consensus_queue.cc:237] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.698846 3392 raft_consensus.cc:491] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.698958 3392 raft_consensus.cc:513] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.699501 3392 leader_election.cc:290] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.700523 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c83f45bd3d469ca55b3d9a5810d27d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:52.701016 710 leader_election.cc:343] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:52.702543 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9791295b4d4b49d796e5c6552f72adbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.704995 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2fe642ca7f64d9fa5c351a4854bd044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.709007 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748253309ff2486fbaaa3217076556f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.710492 3178 raft_consensus.cc:491] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.710628 3178 raft_consensus.cc:513] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.711199 3178 leader_election.cc:290] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
W20250112 20:51:52.712209 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:52.713052 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2881ca71d3949ee82b476c765387a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.717028 1164 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.718593 1164 tablet_bootstrap.cc:654] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.725620 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.728992 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2f3e0b9f05455090bddaa15fe0a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.731829 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c7a57ea399194a45bc8990a029550440
I20250112 20:51:52.733093 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a57ea399194a45bc8990a029550440 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.734408 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 52a4ec67af024fe0be2c06e35117acc6
I20250112 20:51:52.735662 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.736303 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 5ccb593413124921af94d0b855789b52
I20250112 20:51:52.736461 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a57ea399194a45bc8990a029550440 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.736902 892 tablet_service.cc:1514] Processing DeleteTablet for tablet c7a57ea399194a45bc8990a029550440 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.738633 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccb593413124921af94d0b855789b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.738759 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccb593413124921af94d0b855789b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.739065 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccb593413124921af94d0b855789b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.739323 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:52.750792 847 leader_election.cc:343] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: 953d2ec8a7564c0bbcf0166487dd1746
I20250112 20:51:52.750998 847 leader_election.cc:304] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:52.751264 3178 raft_consensus.cc:2743] T 953d2ec8a7564c0bbcf0166487dd1746 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:52.761011 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a4ec67af024fe0be2c06e35117acc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.765070 3426 tablet.cc:1721] T dde64da3dd084a4785986c091852cbc6 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:52.689682 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 988) took 3840 ms. Trace:
I20250112 20:51:52.779752 2447 rpcz_store.cc:276] 0112 20:51:48.849128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.849181 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:52.568043 (+3718862us) ts_tablet_manager.cc:1145] Deleting tablet 9989f3dc61df4f75b9bdb85ba3ca3f37
0112 20:51:52.568054 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.584121 (+ 16067us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.647891 (+ 63770us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.689662 (+ 41771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3718747}
I20250112 20:51:52.780433 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9989f3dc61df4f75b9bdb85ba3ca3f37 (table default.Table_26eeb9cb2a0249928bf1dbc292e71d20 [id=84350f6ee4914832aa5cc23c40a9594c]) successfully deleted
I20250112 20:51:52.781414 904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.785279 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 30a0ff8948eb4f0e8998e75d8d16f0eb
I20250112 20:51:52.786113 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:52.786952 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.787632 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e1ffb6b5445f4fedb10499f1f7a32e3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.788282 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ffb6b5445f4fedb10499f1f7a32e3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.789028 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d3bb4d64af5a47b6b91b7fbb8896c065"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.789444 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3bb4d64af5a47b6b91b7fbb8896c065 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.791472 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c83f45bd3d469ca55b3d9a5810d27d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:52.791673 901 raft_consensus.cc:2463] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:52.792183 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.793072 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_60265b7c7f4b48b7929462845b27fa6f" } new_table_name: "default.Table_06a839a7eee64f6889596bab4fe192d8" modify_external_catalogs: true
I20250112 20:51:52.793462 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_93ca51498c9a41faa2de22068cdb8156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.793825 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93ca51498c9a41faa2de22068cdb8156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.795519 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e66533184b3d4477be0b0e3ac0da4b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.795904 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e66533184b3d4477be0b0e3ac0da4b60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.796550 1164 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.796701 1164 ts_tablet_manager.cc:1397] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.001s
I20250112 20:51:52.797706 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_75d83b84b05842c28d2a5c7aa69860c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:52.797773 1164 raft_consensus.cc:357] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.797973 1164 raft_consensus.cc:383] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.798031 1164 raft_consensus.cc:738] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:52.798096 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75d83b84b05842c28d2a5c7aa69860c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.798216 1164 consensus_queue.cc:260] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.798756 1164 ts_tablet_manager.cc:1428] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:52.798959 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f8730673ed4e8c93ef7ec4d88a2bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.799080 2447 tablet_replica.cc:331] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.799304 2447 raft_consensus.cc:2238] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:52.799618 2447 raft_consensus.cc:2267] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.800204 2447 ts_tablet_manager.cc:1905] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.800550 448 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:52.807178 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 725e56fa8ac343e5b1ec1214da02b7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.807370 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_854e574d01624d3aa32c5cfef84ae0e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.807789 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_854e574d01624d3aa32c5cfef84ae0e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.806919 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_dface6a2477f47e893a51ed847320520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.814047 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dface6a2477f47e893a51ed847320520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:52.817306 983 leader_election.cc:343] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 725e56fa8ac343e5b1ec1214da02b7d4
W20250112 20:51:52.817651 983 leader_election.cc:343] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b37b00c49f234dc89d63f644febc23fb
I20250112 20:51:52.817735 983 leader_election.cc:304] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:52.818089 983 leader_election.cc:343] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 725e56fa8ac343e5b1ec1214da02b7d4
I20250112 20:51:52.818524 3130 raft_consensus.cc:3054] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.821489 450 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:52.822214 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_04069d4b7bad4757988c3e7e8ab02b44" } modify_external_catalogs: true
I20250112 20:51:52.822360 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:52.827561 1185 ts_tablet_manager.cc:1918] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.827684 1185 log.cc:1198] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/104d6b4effb84314a6b2b9db883b4366
I20250112 20:51:52.828107 1185 ts_tablet_manager.cc:1939] T 104d6b4effb84314a6b2b9db883b4366 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.828678 1163 tablet_bootstrap.cc:492] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.829041 3130 raft_consensus.cc:2743] T b37b00c49f234dc89d63f644febc23fb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:52.829099 1163 ts_tablet_manager.cc:1397] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.237s user 0.004s sys 0.004s
I20250112 20:51:52.830050 1163 raft_consensus.cc:357] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.830296 1163 raft_consensus.cc:383] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.830415 1163 raft_consensus.cc:738] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.830693 1163 consensus_queue.cc:260] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.831257 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 725e56fa8ac343e5b1ec1214da02b7d4 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:52.831290 1163 ts_tablet_manager.cc:1428] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:52.831558 1163 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:52.832134 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9db38abce01448dabb432c5b84b46dff consensus state has no leader
I20250112 20:51:52.835908 1163 tablet_bootstrap.cc:654] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.839862 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9be1f065ec4605a0a3fe69770793d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:41 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:52.851389 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_ad29dfc257d84f3fb66f45aa86e14ece [id=97c3a7bf3d874bec9dba134fb054d393] alter complete (version 1)
I20250112 20:51:52.852440 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:52.858827 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2546fb715e33431b915fe26d967b24c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.859333 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2546fb715e33431b915fe26d967b24c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.860441 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_df552fa151414d49923bc94bb418fc9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:52.861229 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df552fa151414d49923bc94bb418fc9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.862084 2447 ts_tablet_manager.cc:1918] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.862545 2447 log.cc:1198] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/98547168378947df885c601ccdc85e57
I20250112 20:51:52.867174 1178 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.867576 2447 ts_tablet_manager.cc:1939] T 98547168378947df885c601ccdc85e57 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:52.867895 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 987) took 4018 ms. Trace:
I20250112 20:51:52.867986 2447 rpcz_store.cc:276] 0112 20:51:48.848994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:48.849368 (+ 374us) service_pool.cc:225] Handling call
0112 20:51:52.799000 (+3949632us) ts_tablet_manager.cc:1145] Deleting tablet 98547168378947df885c601ccdc85e57
0112 20:51:52.799039 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.833566 (+ 34527us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.861978 (+ 28412us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.867857 (+ 5879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3949490}
I20250112 20:51:52.868353 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 98547168378947df885c601ccdc85e57 (table default.Table_8fc6da3983cb406e8e10ecf3f3fd239b [id=e43c79dc98b6435caf4f5eb6d8c7da7b]) successfully deleted
I20250112 20:51:52.868501 1178 tablet_bootstrap.cc:654] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.866554 1163 tablet_bootstrap.cc:492] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.868952 1163 ts_tablet_manager.cc:1397] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20250112 20:51:52.869604 1163 raft_consensus.cc:357] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.869787 1163 raft_consensus.cc:383] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.869841 1163 raft_consensus.cc:738] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.870064 1163 consensus_queue.cc:260] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.870545 1163 ts_tablet_manager.cc:1428] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:52.871446 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11be9d5cff2043df81ecfc6255c40004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.872857 1163 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:52.872663 710 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250112 20:51:52.875337 710 leader_election.cc:304] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:52.875700 3265 raft_consensus.cc:2743] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:52.875938 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b51849dd2647508ec6ec06503ff850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.877786 1186 tablet_replica.cc:331] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.877979 1186 raft_consensus.cc:2238] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.878110 1186 raft_consensus.cc:2267] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.878545 1186 ts_tablet_manager.cc:1905] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:52.881042 710 leader_election.cc:343] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: db9bab2477a94c169817d9de8aaad1c0
W20250112 20:51:52.881542 710 leader_election.cc:343] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: db9bab2477a94c169817d9de8aaad1c0
I20250112 20:51:52.882503 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1002) took 3796 ms. Trace:
I20250112 20:51:52.882586 2447 rpcz_store.cc:276] 0112 20:51:49.085882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.087671 (+ 1789us) service_pool.cc:225] Handling call
0112 20:51:52.882372 (+3794701us) ts_tablet_manager.cc:1145] Deleting tablet 0866871b9e1542e8ab30505c68f7dea9
0112 20:51:52.882384 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.882483 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3794543}
I20250112 20:51:52.882918 1163 tablet_bootstrap.cc:654] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.883935 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1003) took 3613 ms. Trace:
I20250112 20:51:52.884009 2447 rpcz_store.cc:276] 0112 20:51:49.270897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.336641 (+ 65744us) service_pool.cc:225] Handling call
0112 20:51:52.883803 (+3547162us) ts_tablet_manager.cc:1145] Deleting tablet 532ccaeaf38c41a6ab26f65411d31418
0112 20:51:52.883811 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.883916 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3517610}
I20250112 20:51:52.884403 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1011) took 3470 ms. Trace:
I20250112 20:51:52.884502 2447 rpcz_store.cc:276] 0112 20:51:49.413508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.418126 (+ 4618us) service_pool.cc:225] Handling call
0112 20:51:52.884316 (+3466190us) ts_tablet_manager.cc:1145] Deleting tablet 7959f3ed319f4078ab747d490dce7fe1
0112 20:51:52.884323 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.884390 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465930}
I20250112 20:51:52.884639 1186 ts_tablet_manager.cc:1918] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:52.884725 1186 log.cc:1198] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/bf66492d01a64eb1b842acc279b8ee09
I20250112 20:51:52.885010 1186 ts_tablet_manager.cc:1939] T bf66492d01a64eb1b842acc279b8ee09 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.885357 2447 tablet_replica.cc:331] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:52.885458 1186 tablet_replica.cc:331] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.885635 1186 raft_consensus.cc:2238] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.885650 2447 raft_consensus.cc:2238] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.885780 1186 raft_consensus.cc:2267] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.885937 2447 raft_consensus.cc:2267] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.886250 1186 ts_tablet_manager.cc:1905] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.886560 2447 ts_tablet_manager.cc:1905] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:52.889281 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 0866871b9e1542e8ab30505c68f7dea9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0866871b9e1542e8ab30505c68f7dea9 already in progress: creating tablet
W20250112 20:51:52.889497 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 532ccaeaf38c41a6ab26f65411d31418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 532ccaeaf38c41a6ab26f65411d31418 already in progress: creating tablet
W20250112 20:51:52.889655 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7959f3ed319f4078ab747d490dce7fe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7959f3ed319f4078ab747d490dce7fe1 already in progress: creating tablet
I20250112 20:51:52.890096 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_eece5bebd4c54ee784339d7cb43daf6e" } new_table_name: "default.Table_fae781a8a94d42c9a6ede1c013980e8c" modify_external_catalogs: true
I20250112 20:51:52.890288 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet ab9be1f065ec4605a0a3fe69770793d0 (table default.Table_9dde7822fdb743bda1e94189dd58b5ad [id=66b8111f4fbe4fc1a46a09a1c89b7f58]) successfully deleted
I20250112 20:51:52.890468 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet bf66492d01a64eb1b842acc279b8ee09 (table default.Table_ee975b071729460b863016012a403cb4 [id=e1d57fdfdc434d36bb69c54d5c974f41]) successfully deleted
W20250112 20:51:52.890630 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 52a4ec67af024fe0be2c06e35117acc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52a4ec67af024fe0be2c06e35117acc6 already in progress: creating tablet
I20250112 20:51:52.891645 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2b37091fbda64e07b5641542b2961d72" } new_table_name: "default.Table_cb59ccaa39f84b90b998e2cee469100b" modify_external_catalogs: true
I20250112 20:51:52.891445 710 leader_election.cc:304] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:52.892681 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3300f1667af244cfb434fd02511942b6" } new_table_name: "default.Table_ba62c67a244a4f66b09888d3b055c85b" modify_external_catalogs: true
I20250112 20:51:52.899233 1186 ts_tablet_manager.cc:1918] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 20:51:52.899348 1186 log.cc:1198] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/c7a57ea399194a45bc8990a029550440
I20250112 20:51:52.899690 1186 ts_tablet_manager.cc:1939] T c7a57ea399194a45bc8990a029550440 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.900120 1186 tablet_replica.cc:331] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:52.900295 1186 raft_consensus.cc:2238] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.900453 1186 raft_consensus.cc:2267] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.903029 1164 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.904461 1164 tablet_bootstrap.cc:654] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.907534 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1053) took 3554 ms. Trace:
I20250112 20:51:52.907629 1185 rpcz_store.cc:276] 0112 20:51:49.353275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.389328 (+ 36053us) service_pool.cc:225] Handling call
0112 20:51:52.611888 (+3222560us) ts_tablet_manager.cc:1145] Deleting tablet 104d6b4effb84314a6b2b9db883b4366
0112 20:51:52.648896 (+ 37008us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.715446 (+ 66550us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.827454 (+112008us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.907514 (+ 80060us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2824,"tablet-delete.queue_time_us":3222378}
I20250112 20:51:52.908136 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1061) took 3504 ms. Trace:
I20250112 20:51:52.908214 1185 rpcz_store.cc:276] 0112 20:51:49.404027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.404077 (+ 50us) service_pool.cc:225] Handling call
0112 20:51:52.907987 (+3503910us) ts_tablet_manager.cc:1145] Deleting tablet 7959f3ed319f4078ab747d490dce7fe1
0112 20:51:52.908000 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.908113 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3503696}
I20250112 20:51:52.908547 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1088) took 3028 ms. Trace:
I20250112 20:51:52.908618 1185 rpcz_store.cc:276] 0112 20:51:49.880234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.880294 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:52.908454 (+3028160us) ts_tablet_manager.cc:1145] Deleting tablet 7623bebb539e485aa508e95a35225979
0112 20:51:52.908465 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.908534 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027978}
I20250112 20:51:52.908949 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1089) took 3024 ms. Trace:
I20250112 20:51:52.909021 1185 rpcz_store.cc:276] 0112 20:51:49.884186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.884258 (+ 72us) service_pool.cc:225] Handling call
0112 20:51:52.908865 (+3024607us) ts_tablet_manager.cc:1145] Deleting tablet c3812f8aa9ba422f85f331648a1ec02e
0112 20:51:52.908873 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.908936 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024416}
I20250112 20:51:52.909345 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1090) took 3021 ms. Trace:
I20250112 20:51:52.909417 1185 rpcz_store.cc:276] 0112 20:51:49.887491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.887534 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:52.909257 (+3021723us) ts_tablet_manager.cc:1145] Deleting tablet c90c8dd4dfb14507accb82828be512f0
0112 20:51:52.909264 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.909333 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021577}
I20250112 20:51:52.909674 1185 tablet_replica.cc:331] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:52.909878 1185 raft_consensus.cc:2238] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:52.910043 1185 raft_consensus.cc:2267] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.910440 1186 ts_tablet_manager.cc:1905] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.910548 1185 ts_tablet_manager.cc:1905] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.914831 3265 raft_consensus.cc:2798] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.914983 3265 raft_consensus.cc:491] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.915105 3265 raft_consensus.cc:3054] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.917726 1163 tablet_bootstrap.cc:492] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:52.917972 1163 ts_tablet_manager.cc:1397] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250112 20:51:52.918754 1163 raft_consensus.cc:357] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.919178 1163 raft_consensus.cc:383] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.919296 1163 raft_consensus.cc:738] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.919559 1163 consensus_queue.cc:260] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.920087 1163 ts_tablet_manager.cc:1428] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:52.920380 1163 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:52.921697 847 consensus_peers.cc:487] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:52.921757 1164 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.921871 1164 ts_tablet_manager.cc:1397] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20250112 20:51:52.922634 1164 raft_consensus.cc:357] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.922876 1164 raft_consensus.cc:383] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.922994 1164 raft_consensus.cc:738] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.923235 1164 consensus_queue.cc:260] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.923470 3265 raft_consensus.cc:513] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.923821 1164 ts_tablet_manager.cc:1428] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:52.924077 1178 tablet_bootstrap.cc:492] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.924206 1178 ts_tablet_manager.cc:1397] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250112 20:51:52.924834 1178 raft_consensus.cc:357] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.924997 1178 raft_consensus.cc:383] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.925041 1178 raft_consensus.cc:738] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.925220 1178 consensus_queue.cc:260] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.925668 1178 ts_tablet_manager.cc:1428] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:52.925891 1178 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.924185 3265 leader_election.cc:290] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.922742 1185 ts_tablet_manager.cc:1918] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.926934 1164 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.926972 1185 log.cc:1198] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e0fd4177f9394a5a82d4f5e02369b3b6
I20250112 20:51:52.927153 3373 raft_consensus.cc:491] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.927263 3373 raft_consensus.cc:513] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.927588 1185 ts_tablet_manager.cc:1939] T e0fd4177f9394a5a82d4f5e02369b3b6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.927848 3373 leader_election.cc:290] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:52.928159 1164 tablet_bootstrap.cc:654] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:52.928570 983 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ab9be1f065ec4605a0a3fe69770793d0
I20250112 20:51:52.929373 1178 tablet_bootstrap.cc:654] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.929646 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1099) took 3029 ms. Trace:
I20250112 20:51:52.929725 1185 rpcz_store.cc:276] 0112 20:51:49.899892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.901771 (+ 1879us) service_pool.cc:225] Handling call
0112 20:51:52.909633 (+3007862us) ts_tablet_manager.cc:1145] Deleting tablet e0fd4177f9394a5a82d4f5e02369b3b6
0112 20:51:52.909644 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.918563 (+ 8919us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.922653 (+ 4090us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.929630 (+ 6977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007637}
I20250112 20:51:52.930097 1185 tablet_replica.cc:331] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:52.930282 1185 raft_consensus.cc:2238] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:52.930577 1185 raft_consensus.cc:2267] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:52.931058 1185 ts_tablet_manager.cc:1905] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:52.931433 1186 ts_tablet_manager.cc:1918] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:52.931542 1186 log.cc:1198] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/5ccb593413124921af94d0b855789b52
I20250112 20:51:52.931948 1186 ts_tablet_manager.cc:1939] T 5ccb593413124921af94d0b855789b52 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:52.934064 1163 tablet_bootstrap.cc:654] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.944245 3427 tablet.cc:1721] T dde64da3dd084a4785986c091852cbc6 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:52.955986 3304 consensus_queue.cc:1035] T a20d85a4be874b0b90c528a631a29aed P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:52.957643 900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c83f45bd3d469ca55b3d9a5810d27d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:52.957654 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c83f45bd3d469ca55b3d9a5810d27d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:52.957814 900 raft_consensus.cc:3054] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.957814 1041 raft_consensus.cc:3054] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.960438 1041 raft_consensus.cc:2463] T 07c83f45bd3d469ca55b3d9a5810d27d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.960438 900 raft_consensus.cc:2463] T 07c83f45bd3d469ca55b3d9a5810d27d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:52.965178 710 leader_election.cc:304] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:52.965516 3391 raft_consensus.cc:2798] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.965974 3391 raft_consensus.cc:695] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:52.966404 3391 consensus_queue.cc:237] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.969513 1178 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:52.970712 1178 ts_tablet_manager.cc:1397] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.005s
I20250112 20:51:52.972216 1178 raft_consensus.cc:357] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.972622 1178 raft_consensus.cc:383] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.972757 1178 raft_consensus.cc:738] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.973173 1178 consensus_queue.cc:260] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:52.976958 1178 ts_tablet_manager.cc:1428] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:52.978196 1178 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:52.985877 1164 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:52.986035 1164 ts_tablet_manager.cc:1397] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.012s
I20250112 20:51:52.987190 1164 raft_consensus.cc:357] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.987390 1164 raft_consensus.cc:383] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.987448 1164 raft_consensus.cc:738] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.987671 1164 consensus_queue.cc:260] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:52.988195 1164 ts_tablet_manager.cc:1428] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:52.988451 1164 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:52.989742 1164 tablet_bootstrap.cc:654] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.990681 1178 tablet_bootstrap.cc:654] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.995355 1185 ts_tablet_manager.cc:1918] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:52.995465 1185 log.cc:1198] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/61b74aaf7f5b4c87acbefaa4df5c10d2
I20250112 20:51:52.995786 1185 ts_tablet_manager.cc:1939] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:52.905505 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet c7a57ea399194a45bc8990a029550440 (table default.Table_3591211d8ed5415b820beae2848ac1f5 [id=8133cc129a404fb8aca53d1fdb1c7fef]) successfully deleted
W20250112 20:51:53.000000 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 94495.5us
I20250112 20:51:53.002014 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_28f569ded7ee456cb79ce58b8adb809d" } modify_external_catalogs: true
I20250112 20:51:53.002187 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_28f569ded7ee456cb79ce58b8adb809d" } modify_external_catalogs: true
I20250112 20:51:53.002269 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 104d6b4effb84314a6b2b9db883b4366 (table default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]) successfully deleted
W20250112 20:51:53.002493 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 7959f3ed319f4078ab747d490dce7fe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7959f3ed319f4078ab747d490dce7fe1 already in progress: creating tablet
W20250112 20:51:53.002676 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 7623bebb539e485aa508e95a35225979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7623bebb539e485aa508e95a35225979 already in progress: creating tablet
I20250112 20:51:53.002724 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_394111b499bd4b8ca206c37121264bb4" } new_table_name: "default.Table_455de0f0fd444c0c9d82d9547906a8e6" modify_external_catalogs: true
I20250112 20:51:53.003031 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_162179129dc94078b87f0bf9820e6253" } modify_external_catalogs: true
I20250112 20:51:53.003147 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_162179129dc94078b87f0bf9820e6253" } modify_external_catalogs: true
W20250112 20:51:53.002830 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c3812f8aa9ba422f85f331648a1ec02e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3812f8aa9ba422f85f331648a1ec02e already in progress: creating tablet
W20250112 20:51:53.003650 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet c90c8dd4dfb14507accb82828be512f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c90c8dd4dfb14507accb82828be512f0 already in progress: creating tablet
I20250112 20:51:53.003901 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e0fd4177f9394a5a82d4f5e02369b3b6 (table default.Table_cba467c888b84da6964f049247721310 [id=c0f4f9a776794374b5cfc063becc2876]) successfully deleted
I20250112 20:51:53.003962 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e553dd8387024dbca022b562a69871a5" } modify_external_catalogs: true
I20250112 20:51:53.004081 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e553dd8387024dbca022b562a69871a5" } modify_external_catalogs: true
I20250112 20:51:53.004552 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f5df1b9a2b9947849b4948b8dcdfdc09" } modify_external_catalogs: true
I20250112 20:51:53.004690 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f5df1b9a2b9947849b4948b8dcdfdc09" } modify_external_catalogs: true
I20250112 20:51:53.005043 2447 ts_tablet_manager.cc:1918] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.005141 2447 log.cc:1198] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/e8bea4d83ea644a0930a29f61e7a5fc8
I20250112 20:51:53.005175 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4d8793ff73b749f1a8e278bd1edcba89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.005419 2447 ts_tablet_manager.cc:1939] T e8bea4d83ea644a0930a29f61e7a5fc8 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:53.005664 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d8793ff73b749f1a8e278bd1edcba89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.007104 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1014) took 3367 ms. Trace:
I20250112 20:51:53.008435 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 5ccb593413124921af94d0b855789b52 (table default.Table_e5949130bc9c488da66e7ebeeaf6cc8c [id=55f399c9f7d34345a15b18ed312bf623]) successfully deleted
W20250112 20:51:53.009218 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 30a0ff8948eb4f0e8998e75d8d16f0eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a0ff8948eb4f0e8998e75d8d16f0eb already in progress: creating tablet
I20250112 20:51:53.009409 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:53.010329 2447 rpcz_store.cc:276] 0112 20:51:49.639261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.648823 (+ 9562us) service_pool.cc:225] Handling call
0112 20:51:52.885199 (+3236376us) ts_tablet_manager.cc:1145] Deleting tablet e8bea4d83ea644a0930a29f61e7a5fc8
0112 20:51:52.885211 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.924704 (+ 39493us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.004943 (+ 80239us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.007088 (+ 2145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3236001}
I20250112 20:51:53.010767 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1043) took 3089 ms. Trace:
I20250112 20:51:53.010789 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet e8bea4d83ea644a0930a29f61e7a5fc8 (table default.Table_92f355d4058f40b6b0979ff3f3f1af14 [id=2049ad7ffcee4affa56490ef29e949da]) successfully deleted
I20250112 20:51:53.010869 2447 rpcz_store.cc:276] 0112 20:51:49.920896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.923116 (+ 2220us) service_pool.cc:225] Handling call
0112 20:51:53.010652 (+3087536us) ts_tablet_manager.cc:1145] Deleting tablet 961a72a9d8a643cea743f3f021bd5a03
0112 20:51:53.010660 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.010752 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3087375}
I20250112 20:51:53.009173 1163 tablet_bootstrap.cc:492] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.011158 1163 ts_tablet_manager.cc:1397] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.000s
I20250112 20:51:53.011884 1163 raft_consensus.cc:357] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.012097 1163 raft_consensus.cc:383] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.012176 1163 raft_consensus.cc:738] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:53.012288 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d94c67881e954c0ebecf5c31de665ac0 consensus state has no leader
I20250112 20:51:53.012406 1163 consensus_queue.cc:260] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.013058 1163 ts_tablet_manager.cc:1428] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:53.013325 1163 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.016151 1163 tablet_bootstrap.cc:654] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.017730 3438 tablet.cc:1721] T dde64da3dd084a4785986c091852cbc6 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.018707 3178 raft_consensus.cc:491] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.018841 3178 raft_consensus.cc:513] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.019392 3178 leader_election.cc:290] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.020663 3391 raft_consensus.cc:491] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.020815 3391 raft_consensus.cc:513] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.021363 3391 leader_election.cc:290] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:53.024255 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 961a72a9d8a643cea743f3f021bd5a03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961a72a9d8a643cea743f3f021bd5a03 already in progress: creating tablet
I20250112 20:51:53.024623 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1040) took 3104 ms. Trace:
I20250112 20:51:53.024708 2447 rpcz_store.cc:276] 0112 20:51:49.920476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.945420 (+ 24944us) service_pool.cc:225] Handling call
0112 20:51:53.024504 (+3079084us) ts_tablet_manager.cc:1145] Deleting tablet c3812f8aa9ba422f85f331648a1ec02e
0112 20:51:53.024515 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.024606 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3077892}
W20250112 20:51:53.025096 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c3812f8aa9ba422f85f331648a1ec02e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3812f8aa9ba422f85f331648a1ec02e already in progress: creating tablet
I20250112 20:51:53.031651 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1044) took 3084 ms. Trace:
I20250112 20:51:53.031741 2447 rpcz_store.cc:276] 0112 20:51:49.947040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.952948 (+ 5908us) service_pool.cc:225] Handling call
0112 20:51:53.031522 (+3078574us) ts_tablet_manager.cc:1145] Deleting tablet a5819f1c6329433183492855c3b08476
0112 20:51:53.031531 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.031634 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3078296}
I20250112 20:51:53.032032 2447 tablet_replica.cc:331] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
W20250112 20:51:53.032207 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a5819f1c6329433183492855c3b08476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5819f1c6329433183492855c3b08476 already in progress: creating tablet
I20250112 20:51:53.032235 2447 raft_consensus.cc:2238] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.032387 2447 raft_consensus.cc:2267] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.032932 2447 ts_tablet_manager.cc:1905] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.033934 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 940756f8867b4cde91fb0e89e6069265 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:53.034132 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d167065029a46b791d413db8a2c4bcc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.034307 1044 raft_consensus.cc:2463] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:53.037106 710 leader_election.cc:304] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.037515 3391 raft_consensus.cc:2798] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.037607 3391 raft_consensus.cc:491] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.037662 3391 raft_consensus.cc:3054] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.040199 3391 raft_consensus.cc:513] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.040802 3391 leader_election.cc:290] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.035702 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_2f77351ffe3d4bb995bee57e885a76de" } new_table_name: "default.Table_80416acb7463460f9ebe7f7b5245960e" modify_external_catalogs: true
I20250112 20:51:53.047099 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d167065029a46b791d413db8a2c4bcc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:53.047278 1044 raft_consensus.cc:3054] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:53.051432 710 leader_election.cc:343] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 5d167065029a46b791d413db8a2c4bcc
W20250112 20:51:53.051656 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89e2d166a2a24e8ca4830936171dd36d consensus state has no leader
I20250112 20:51:53.053372 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:53.053556 766 raft_consensus.cc:2463] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:53.054013 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:53.054219 771 raft_consensus.cc:2463] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:53.052654 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_c65b1f79389e468fb8793d95d7248128"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.057796 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c65b1f79389e468fb8793d95d7248128 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.062819 1044 raft_consensus.cc:2463] T 5d167065029a46b791d413db8a2c4bcc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:53.069686 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1097) took 3170 ms. Trace:
I20250112 20:51:53.069840 1185 rpcz_store.cc:276] 0112 20:51:49.899297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.911122 (+ 11825us) service_pool.cc:225] Handling call
0112 20:51:52.930059 (+3018937us) ts_tablet_manager.cc:1145] Deleting tablet 61b74aaf7f5b4c87acbefaa4df5c10d2
0112 20:51:52.930067 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:52.992319 (+ 62252us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:52.995265 (+ 2946us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.069659 (+ 74394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3018569}
I20250112 20:51:53.070353 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (table default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]) successfully deleted
I20250112 20:51:53.080698 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.081094 1047 raft_consensus.cc:2463] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 0.
I20250112 20:51:53.081602 847 leader_election.cc:304] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:53.081990 3178 raft_consensus.cc:2798] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.082077 3178 raft_consensus.cc:491] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.082147 3178 raft_consensus.cc:3054] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.085505 3178 raft_consensus.cc:513] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.086704 3178 leader_election.cc:290] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.086720 1185 tablet_replica.cc:331] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:53.086930 1185 raft_consensus.cc:2238] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:53.087035 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:53.087244 1044 raft_consensus.cc:3054] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.087271 1185 raft_consensus.cc:2267] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.087787 1185 ts_tablet_manager.cc:1905] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.090060 1044 raft_consensus.cc:2463] T d738720dd7414764b4d1191cdc37895d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
I20250112 20:51:53.090553 847 leader_election.cc:304] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:53.092176 3178 raft_consensus.cc:2798] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.092761 3178 raft_consensus.cc:695] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Running, Role: LEADER
I20250112 20:51:53.093175 3178 consensus_queue.cc:237] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.095376 983 leader_election.cc:304] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.095821 3130 raft_consensus.cc:2798] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.095883 3130 raft_consensus.cc:491] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.095974 3130 raft_consensus.cc:3054] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.097056 1163 tablet_bootstrap.cc:492] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
W20250112 20:51:53.097038 710 leader_election.cc:343] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 5d167065029a46b791d413db8a2c4bcc
I20250112 20:51:53.097190 1163 ts_tablet_manager.cc:1397] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20250112 20:51:53.097770 710 leader_election.cc:304] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.097895 1163 raft_consensus.cc:357] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.098047 1163 raft_consensus.cc:383] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.098096 1163 raft_consensus.cc:738] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.098126 3391 raft_consensus.cc:2798] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.098246 3391 raft_consensus.cc:695] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:53.098341 3130 raft_consensus.cc:513] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.098428 1163 consensus_queue.cc:260] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.098454 3391 consensus_queue.cc:237] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.099012 3130 leader_election.cc:290] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.099179 1163 ts_tablet_manager.cc:1428] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:53.099227 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d738720dd7414764b4d1191cdc37895d" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:53.099367 760 raft_consensus.cc:3054] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.099424 1163 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.102339 760 raft_consensus.cc:2463] T d738720dd7414764b4d1191cdc37895d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ae45957c64469d96dc92d606b5b4b3 in term 1.
W20250112 20:51:53.105484 983 leader_election.cc:343] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: ab9be1f065ec4605a0a3fe69770793d0
I20250112 20:51:53.106981 1178 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.107194 1178 ts_tablet_manager.cc:1397] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.129s user 0.004s sys 0.000s
I20250112 20:51:53.109548 1178 raft_consensus.cc:357] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.109828 1178 raft_consensus.cc:383] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.109966 1178 raft_consensus.cc:738] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.110392 1178 consensus_queue.cc:260] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.111202 1178 ts_tablet_manager.cc:1428] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:53.111563 1178 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.111749 1163 tablet_bootstrap.cc:654] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.114482 1164 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:53.114601 1164 ts_tablet_manager.cc:1397] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.003s
I20250112 20:51:53.116508 1178 tablet_bootstrap.cc:654] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.119940 1164 raft_consensus.cc:357] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.120139 1164 raft_consensus.cc:383] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.120225 1164 raft_consensus.cc:738] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.120447 1164 consensus_queue.cc:260] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.120994 1164 ts_tablet_manager.cc:1428] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 20:51:53.121270 1164 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:53.121752 1178 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.121881 1178 ts_tablet_manager.cc:1397] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250112 20:51:53.122059 2447 ts_tablet_manager.cc:1918] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.122172 2447 log.cc:1198] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/f76dbd7f681b418ca7bb6c6e8f1f1e58
I20250112 20:51:53.122485 2447 ts_tablet_manager.cc:1939] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:53.122634 1178 raft_consensus.cc:357] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.122826 1164 tablet_bootstrap.cc:654] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.122828 1178 raft_consensus.cc:383] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.122877 1178 raft_consensus.cc:738] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.123152 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1046) took 3166 ms. Trace:
I20250112 20:51:53.123121 1178 consensus_queue.cc:260] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.123229 2447 rpcz_store.cc:276] 0112 20:51:49.957112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.957172 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:53.031988 (+3074816us) ts_tablet_manager.cc:1145] Deleting tablet f76dbd7f681b418ca7bb6c6e8f1f1e58
0112 20:51:53.031998 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.086804 (+ 54806us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.121955 (+ 35151us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.123134 (+ 1179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3074578}
I20250112 20:51:53.123667 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (table default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]) successfully deleted
I20250112 20:51:53.123742 1178 ts_tablet_manager.cc:1428] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:53.124625 1178 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.127199 1178 tablet_bootstrap.cc:654] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.128468 1164 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:53.128638 1164 ts_tablet_manager.cc:1397] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250112 20:51:53.130749 1164 raft_consensus.cc:357] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.130935 1164 raft_consensus.cc:383] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.130981 1164 raft_consensus.cc:738] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.131172 1164 consensus_queue.cc:260] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.131640 1164 ts_tablet_manager.cc:1428] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 20:51:53.131913 1164 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:53.133198 1178 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.133363 1178 ts_tablet_manager.cc:1397] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250112 20:51:53.133478 1164 tablet_bootstrap.cc:654] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.134048 1178 raft_consensus.cc:357] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.134282 1178 raft_consensus.cc:383] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.134384 1178 raft_consensus.cc:738] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.134631 1178 consensus_queue.cc:260] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.135241 1178 ts_tablet_manager.cc:1428] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:53.138679 1178 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.140038 900 raft_consensus.cc:1270] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Refusing update from remote peer aefe1db8b19244d891b6bcd8971e9b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:53.140956 1178 tablet_bootstrap.cc:654] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:53.148193 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 89e2d166a2a24e8ca4830936171dd36d (table default.Table_80416acb7463460f9ebe7f7b5245960e [id=6071e3cfbb2147259555da613232515d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 89e2d166a2a24e8ca4830936171dd36d consensus state has no leader
I20250112 20:51:53.148159 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9be1f065ec4605a0a3fe69770793d0" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:53.148344 759 raft_consensus.cc:3054] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.151852 759 raft_consensus.cc:2463] T ab9be1f065ec4605a0a3fe69770793d0 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:53.153976 2447 tablet_replica.cc:331] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:53.154242 2447 raft_consensus.cc:2238] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.155045 2447 raft_consensus.cc:2267] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.155632 2447 ts_tablet_manager.cc:1905] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.158200 3114 raft_consensus.cc:491] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.158321 3114 raft_consensus.cc:513] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.158876 3114 leader_election.cc:290] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.187156 2447 ts_tablet_manager.cc:1918] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:53.187357 2447 log.cc:1198] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/53551931b23d43da89e9193cdf034a63
I20250112 20:51:53.187736 2447 ts_tablet_manager.cc:1939] T 53551931b23d43da89e9193cdf034a63 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:53.188092 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1045) took 3240 ms. Trace:
I20250112 20:51:53.188452 2447 rpcz_store.cc:276] 0112 20:51:49.947166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.958426 (+ 11260us) service_pool.cc:225] Handling call
0112 20:51:53.153927 (+3195501us) ts_tablet_manager.cc:1145] Deleting tablet 53551931b23d43da89e9193cdf034a63
0112 20:51:53.153937 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.183519 (+ 29582us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.187062 (+ 3543us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.188076 (+ 1014us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3195247}
I20250112 20:51:53.188987 2447 tablet_replica.cc:331] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:53.189111 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]) successfully deleted
I20250112 20:51:53.189240 2447 raft_consensus.cc:2238] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.189383 2447 raft_consensus.cc:2267] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.190011 2447 ts_tablet_manager.cc:1905] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:53.190261 710 consensus_peers.cc:487] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.190476 3114 raft_consensus.cc:491] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.190606 3114 raft_consensus.cc:513] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.191185 3114 leader_election.cc:290] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.191262 3265 consensus_queue.cc:1035] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.191496 3426 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.195384 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e2c3f72ab4f439678d166b9b31661" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:53.195600 909 raft_consensus.cc:2463] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:53.196033 3438 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.200089 899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b370f96044305a79ebdb79d5e01d1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:53.200268 899 raft_consensus.cc:2463] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:53.206456 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d9f68cf1f11247f49b9834bfd4f94306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.207011 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f68cf1f11247f49b9834bfd4f94306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.211226 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_a4190ff9da0d40a58ed6dac7cefcb255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.211671 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4190ff9da0d40a58ed6dac7cefcb255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.213596 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 263b370f96044305a79ebdb79d5e01d1 (table default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4]). Expected version 1 got 0
I20250112 20:51:53.213761 453 catalog_manager.cc:5526] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:53.191589 983 leader_election.cc:304] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.217584 3373 raft_consensus.cc:2798] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.217708 3373 raft_consensus.cc:695] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:53.217897 3373 consensus_queue.cc:237] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.219718 756 tablet_service.cc:1467] Processing CreateTablet for tablet f054c0b937424e31983edb576e8d4bc0 (DEFAULT_TABLE table=default.Table_ab8ef713e8a7432fa513234210a4cb6d [id=ead82135d5d14fa6bff838f741a30995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.219905 738 tablet_service.cc:1467] Processing CreateTablet for tablet 998dac7c12164f5f984f28f8d5ccb1df (DEFAULT_TABLE table=default.Table_cf6ee0c97faa43b0ab60b144c37f65e5 [id=0d7a48fcbd93445eb158ec3d8972fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.220129 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f054c0b937424e31983edb576e8d4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.220363 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998dac7c12164f5f984f28f8d5ccb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.219718 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf03511bb694e25aaf523490328e6ec (DEFAULT_TABLE table=default.Table_5c0a48d4b1f44f1e9d75d400ebab7e4d [id=ac0ebea8976d4fe9b0aef11a217eac50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.220907 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf03511bb694e25aaf523490328e6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.223896 710 leader_election.cc:304] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.224169 3121 raft_consensus.cc:2798] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.224238 3121 raft_consensus.cc:491] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.224303 3121 raft_consensus.cc:3054] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.224284 710 leader_election.cc:304] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.224740 3392 raft_consensus.cc:2798] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.224824 3392 raft_consensus.cc:491] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.224884 3392 raft_consensus.cc:3054] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.226791 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 998dac7c12164f5f984f28f8d5ccb1df (DEFAULT_TABLE table=default.Table_cf6ee0c97faa43b0ab60b144c37f65e5 [id=0d7a48fcbd93445eb158ec3d8972fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.227181 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998dac7c12164f5f984f28f8d5ccb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.228689 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf03511bb694e25aaf523490328e6ec (DEFAULT_TABLE table=default.Table_5c0a48d4b1f44f1e9d75d400ebab7e4d [id=ac0ebea8976d4fe9b0aef11a217eac50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.228991 891 tablet_service.cc:1467] Processing CreateTablet for tablet 475bc060fe7c4a41967c25d476a88dd0 (DEFAULT_TABLE table=default.Table_4be3e81d23584e39ac24e2ed93876f47 [id=6fff4b365ad44b36b73af0240daef804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.229014 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f054c0b937424e31983edb576e8d4bc0 (DEFAULT_TABLE table=default.Table_ab8ef713e8a7432fa513234210a4cb6d [id=ead82135d5d14fa6bff838f741a30995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.229166 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e2c3f72ab4f439678d166b9b31661" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.229362 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475bc060fe7c4a41967c25d476a88dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.229034 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf03511bb694e25aaf523490328e6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.229538 745 tablet_service.cc:1467] Processing CreateTablet for tablet ecf82f714563422fbf2803f1b54a8c61 (DEFAULT_TABLE table=default.Table_251238528b1d4607bb3bf46da6d40a56 [id=da18cf0575d54c239ddde00bffab35f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.229573 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf03511bb694e25aaf523490328e6ec (DEFAULT_TABLE table=default.Table_5c0a48d4b1f44f1e9d75d400ebab7e4d [id=ac0ebea8976d4fe9b0aef11a217eac50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.229838 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf82f714563422fbf2803f1b54a8c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.229838 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf03511bb694e25aaf523490328e6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.233027 749 tablet_service.cc:1467] Processing CreateTablet for tablet 475bc060fe7c4a41967c25d476a88dd0 (DEFAULT_TABLE table=default.Table_4be3e81d23584e39ac24e2ed93876f47 [id=6fff4b365ad44b36b73af0240daef804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.233012 884 tablet_service.cc:1467] Processing CreateTablet for tablet 998dac7c12164f5f984f28f8d5ccb1df (DEFAULT_TABLE table=default.Table_cf6ee0c97faa43b0ab60b144c37f65e5 [id=0d7a48fcbd93445eb158ec3d8972fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.233312 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475bc060fe7c4a41967c25d476a88dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.233312 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998dac7c12164f5f984f28f8d5ccb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.233904 879 tablet_service.cc:1467] Processing CreateTablet for tablet ac98192563714ad890d9c2dcb157fdf6 (DEFAULT_TABLE table=default.Table_a72551b18fc4484c89e651cf3af7ff94 [id=ae6c4bc892e4478f8b126979a84ef203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.234186 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac98192563714ad890d9c2dcb157fdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.236537 739 tablet_service.cc:1467] Processing CreateTablet for tablet ac98192563714ad890d9c2dcb157fdf6 (DEFAULT_TABLE table=default.Table_a72551b18fc4484c89e651cf3af7ff94 [id=ae6c4bc892e4478f8b126979a84ef203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.236872 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac98192563714ad890d9c2dcb157fdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.237044 888 tablet_service.cc:1467] Processing CreateTablet for tablet f054c0b937424e31983edb576e8d4bc0 (DEFAULT_TABLE table=default.Table_ab8ef713e8a7432fa513234210a4cb6d [id=ead82135d5d14fa6bff838f741a30995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.237345 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f054c0b937424e31983edb576e8d4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.238983 3121 raft_consensus.cc:513] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.239549 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9b612eb4a5c24b05aff8d6bb6da866d8 (DEFAULT_TABLE table=default.Table_69b0567cbb9e451fab36564e4c6582ba [id=912f105602124776b43d0e63e0e24642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.239660 3121 leader_election.cc:290] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.239821 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b612eb4a5c24b05aff8d6bb6da866d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.240001 885 tablet_service.cc:1467] Processing CreateTablet for tablet 9b612eb4a5c24b05aff8d6bb6da866d8 (DEFAULT_TABLE table=default.Table_69b0567cbb9e451fab36564e4c6582ba [id=912f105602124776b43d0e63e0e24642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.240319 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b612eb4a5c24b05aff8d6bb6da866d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.240998 889 tablet_service.cc:1467] Processing CreateTablet for tablet ecf82f714563422fbf2803f1b54a8c61 (DEFAULT_TABLE table=default.Table_251238528b1d4607bb3bf46da6d40a56 [id=da18cf0575d54c239ddde00bffab35f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.241315 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf82f714563422fbf2803f1b54a8c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.242487 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 4147bd0534264bf99481a95b8e5120ff
I20250112 20:51:53.244678 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_971f4e13c7144ff0b72e5b6e65629056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.245025 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 940756f8867b4cde91fb0e89e6069265
I20250112 20:51:53.245242 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f054c0b937424e31983edb576e8d4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:53.245254 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_971f4e13c7144ff0b72e5b6e65629056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.245386 449 catalog_manager.cc:6596] ReplaceTablet: tablet 940756f8867b4cde91fb0e89e6069265 deleted and replaced by tablet 9a9393e5cc434ec19bb907a6331e9d26
I20250112 20:51:53.245671 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2627bae62eb34d008e734cba137ae9b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.246106 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2627bae62eb34d008e734cba137ae9b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.247834 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 4147bd0534264bf99481a95b8e5120ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.248044 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 940756f8867b4cde91fb0e89e6069265 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a9393e5cc434ec19bb907a6331e9d26) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.248557 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 475bc060fe7c4a41967c25d476a88dd0 (DEFAULT_TABLE table=default.Table_4be3e81d23584e39ac24e2ed93876f47 [id=6fff4b365ad44b36b73af0240daef804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.248883 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475bc060fe7c4a41967c25d476a88dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.252385 3392 raft_consensus.cc:513] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.253484 3392 leader_election.cc:290] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.253626 3392 raft_consensus.cc:491] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.253641 1013 tablet_service.cc:1467] Processing CreateTablet for tablet ac98192563714ad890d9c2dcb157fdf6 (DEFAULT_TABLE table=default.Table_a72551b18fc4484c89e651cf3af7ff94 [id=ae6c4bc892e4478f8b126979a84ef203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.253721 3392 raft_consensus.cc:513] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.253942 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac98192563714ad890d9c2dcb157fdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.254196 3392 leader_election.cc:290] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.257341 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4147bd0534264bf99481a95b8e5120ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.257406 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9b612eb4a5c24b05aff8d6bb6da866d8 (DEFAULT_TABLE table=default.Table_69b0567cbb9e451fab36564e4c6582ba [id=912f105602124776b43d0e63e0e24642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.257700 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b612eb4a5c24b05aff8d6bb6da866d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.258031 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 940756f8867b4cde91fb0e89e6069265 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a9393e5cc434ec19bb907a6331e9d26) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.258910 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ecf82f714563422fbf2803f1b54a8c61 (DEFAULT_TABLE table=default.Table_251238528b1d4607bb3bf46da6d40a56 [id=da18cf0575d54c239ddde00bffab35f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.259594 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf82f714563422fbf2803f1b54a8c61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.266880 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4147bd0534264bf99481a95b8e5120ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.272408 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f669f2f5bed64a6280189cdee98bc64d
I20250112 20:51:53.272588 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 1b8fe3004a234c83a4dc76b9ed815d30
I20250112 20:51:53.273093 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2e8a11571feb4678a59c17ef38959d7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.273262 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_b216fe36f3134924b3e2adf44426475a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.273627 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e8a11571feb4678a59c17ef38959d7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:53.273690 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b216fe36f3134924b3e2adf44426475a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.277024 1163 tablet_bootstrap.cc:492] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.277573 1163 ts_tablet_manager.cc:1397] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.178s user 0.003s sys 0.006s
I20250112 20:51:53.278771 3426 tablet.cc:1721] T 61fac2cf43c24c158e340cbc8dd176c3 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.284612 1185 ts_tablet_manager.cc:1918] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.284729 1185 log.cc:1198] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/5f264d84ae4846d1afaaabc41d5789f4
I20250112 20:51:53.285102 1185 ts_tablet_manager.cc:1939] T 5f264d84ae4846d1afaaabc41d5789f4 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:53.285331 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:53.285522 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 4147bd0534264bf99481a95b8e5120ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4147bd0534264bf99481a95b8e5120ff already in progress: creating tablet
I20250112 20:51:53.285912 1163 raft_consensus.cc:357] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.287120 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.287349 1163 raft_consensus.cc:383] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.287460 1163 raft_consensus.cc:738] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.287617 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f669f2f5bed64a6280189cdee98bc64d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.287822 1163 consensus_queue.cc:260] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.288381 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.288225 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet f669f2f5bed64a6280189cdee98bc64d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.288728 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd
I20250112 20:51:53.288762 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8
I20250112 20:51:53.289160 450 catalog_manager.cc:6596] ReplaceTablet: tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 deleted and replaced by tablet 6b67d66724e247039cef64b54c4aef07
I20250112 20:51:53.289435 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0ce0b84702c84d5cb03c6761260191db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.289467 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_237a0cfba9424c13a84c153a381d7967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.289948 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_237a0cfba9424c13a84c153a381d7967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.290071 1163 ts_tablet_manager.cc:1428] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 20:51:53.291621 1163 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
W20250112 20:51:53.289959 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ce0b84702c84d5cb03c6761260191db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.293098 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b67d66724e247039cef64b54c4aef07) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:53.293557 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f669f2f5bed64a6280189cdee98bc64d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f669f2f5bed64a6280189cdee98bc64d already in progress: creating tablet
I20250112 20:51:53.293329 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:53.293787 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 1b8fe3004a234c83a4dc76b9ed815d30 because the tablet was not found. No further retry: Not found: Tablet not found: 1b8fe3004a234c83a4dc76b9ed815d30
I20250112 20:51:53.294653 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.295349 1163 tablet_bootstrap.cc:654] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.298439 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1098) took 3398 ms. Trace:
I20250112 20:51:53.300981 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b67d66724e247039cef64b54c4aef07) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.301302 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f669f2f5bed64a6280189cdee98bc64d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.299190 1185 rpcz_store.cc:276] 0112 20:51:49.899601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.911788 (+ 12187us) service_pool.cc:225] Handling call
0112 20:51:53.086657 (+3174869us) ts_tablet_manager.cc:1145] Deleting tablet 5f264d84ae4846d1afaaabc41d5789f4
0112 20:51:53.086672 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.090627 (+ 3955us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.284526 (+193899us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.298186 (+ 13660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3174618}
I20250112 20:51:53.302817 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d396dfcbb5d04b82a52d4eda0ceb5ff2
I20250112 20:51:53.303149 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d9a46ff1902a48529ee58d8c36e0d9e0" } new_table_name: "default.Table_ee725a4c98764d828e0d320138d526d7" modify_external_catalogs: true
I20250112 20:51:53.303501 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3c5be7fd026a4048b1e177fe5956ac0a" } new_table_name: "default.Table_af56f0d63dbc4b06894f55724dc835d6" modify_external_catalogs: true
I20250112 20:51:53.304461 1185 tablet_replica.cc:331] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:53.305737 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_c1535f7854fa43009733487f7b3af237" } new_table_name: "default.Table_91658f5b34f048e3a34302177b4fdce9" modify_external_catalogs: true
I20250112 20:51:53.306551 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.307037 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_78ebfe3d8c41485999992166d9bf2149" } new_table_name: "default.Table_05ba7e2386884bf4906bd06572bfe526" modify_external_catalogs: true
I20250112 20:51:53.307355 2447 ts_tablet_manager.cc:1918] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.307454 2447 log.cc:1198] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:53.307611 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 5f264d84ae4846d1afaaabc41d5789f4 (table default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]) successfully deleted
I20250112 20:51:53.308023 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_351b00da7a0b461e85cb7800408af4f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.306756 738 tablet_service.cc:1514] Processing DeleteTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.307612 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:53.308616 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351b00da7a0b461e85cb7800408af4f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.308149 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fdc1f15839ae4f91a0b63261f416283a" } new_table_name: "default.Table_31cb71e62b274a56b0fac88759a5a963" modify_external_catalogs: true
I20250112 20:51:53.310127 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4550740f517d4eddb1752f880ef3879e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.310832 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4550740f517d4eddb1752f880ef3879e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.311182 1178 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.311390 1178 ts_tablet_manager.cc:1397] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.030s
I20250112 20:51:53.312160 1178 raft_consensus.cc:357] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.312479 1178 raft_consensus.cc:383] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.312587 1178 raft_consensus.cc:738] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.312827 1178 consensus_queue.cc:260] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.313431 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 615d75bc22df45cc876788a266e983cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:39 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.313923 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d167065029a46b791d413db8a2c4bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.314812 1178 ts_tablet_manager.cc:1428] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:53.315297 1178 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.316679 1178 tablet_bootstrap.cc:654] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.317525 1185 raft_consensus.cc:2238] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.319267 1185 raft_consensus.cc:2267] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.320035 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 940756f8867b4cde91fb0e89e6069265 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a9393e5cc434ec19bb907a6331e9d26) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.321699 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:53.321904 1020 tablet_service.cc:1467] Processing CreateTablet for tablet abd40e0cf7ae446182c498e3dedc69ec (DEFAULT_TABLE table=default.Table_a0aadee20eb34f1eb4f122d7d5920824 [id=953c59739f10479eb4960753db269c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.322043 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 0946fe64150e41da957907aa4a6a672f (table default.Table_5709a18905f745a48efbc8020384b017 [id=e203e5c96b52431fa00a10d416d98b8c]). Expected version 1 got 0
I20250112 20:51:53.322242 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:53.322283 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd40e0cf7ae446182c498e3dedc69ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.322430 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 1b1565c7487c428f8f963aca406263fe (table default.Table_bf9df2a098b246ddaf645cbf3426550a [id=e21586dd18394721b1e0e0a5df69259a]). Expected version 4 got 0
I20250112 20:51:53.322945 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 549d92191a594788bf639cd60a6b5142 (table default.Table_cb59ccaa39f84b90b998e2cee469100b [id=3ddc179c72f74f78b0ec696d624429c4]). Expected version 3 got 2
I20250112 20:51:53.323174 451 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet fa2f3389061143509354f37ae93ef0c7 (table default.Table_e99572bd1321493a8d09c4895804e78e [id=8f418af80a3b4cd78aea9984451558ed]). Expected version 2 got 0
I20250112 20:51:53.325076 2447 ts_tablet_manager.cc:1939] T 9880e7d303b54d959bca4e8464a6b30c P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:53.325497 1185 ts_tablet_manager.cc:1905] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.329165 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b370f96044305a79ebdb79d5e01d1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.332726 749 tablet_service.cc:1467] Processing CreateTablet for tablet abd40e0cf7ae446182c498e3dedc69ec (DEFAULT_TABLE table=default.Table_a0aadee20eb34f1eb4f122d7d5920824 [id=953c59739f10479eb4960753db269c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.332981 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1047) took 3372 ms. Trace:
I20250112 20:51:53.333073 2447 rpcz_store.cc:276] 0112 20:51:49.960174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.962503 (+ 2329us) service_pool.cc:225] Handling call
0112 20:51:53.188941 (+3226438us) ts_tablet_manager.cc:1145] Deleting tablet 9880e7d303b54d959bca4e8464a6b30c
0112 20:51:53.188951 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.278541 (+ 89590us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.307238 (+ 28697us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.332967 (+ 25729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226230}
I20250112 20:51:53.333233 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd40e0cf7ae446182c498e3dedc69ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.334030 1163 tablet_bootstrap.cc:492] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.334151 1163 ts_tablet_manager.cc:1397] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.002s
I20250112 20:51:53.334833 1163 raft_consensus.cc:357] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.334988 1163 raft_consensus.cc:383] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.335034 1163 raft_consensus.cc:738] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.335249 1163 consensus_queue.cc:260] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.336279 1163 ts_tablet_manager.cc:1428] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:53.336611 1163 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.336663 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9880e7d303b54d959bca4e8464a6b30c (table default.Table_0cf0e2c39db54427949f7adfbec95e0d [id=537bf8b60c0249efb50df78260ed60c3]) successfully deleted
I20250112 20:51:53.337536 1178 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.337672 1178 ts_tablet_manager.cc:1397] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250112 20:51:53.338263 1178 raft_consensus.cc:357] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.338462 1178 raft_consensus.cc:383] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.338523 1178 raft_consensus.cc:738] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.338996 1163 tablet_bootstrap.cc:654] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.340193 1178 consensus_queue.cc:260] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.340364 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 46cdb418ef204387939ac72a8bfc06ab (DEFAULT_TABLE table=default.Table_59bfceefa0364022aa60152f99bcff6f [id=dd42e2b4ae474dcc93718276325f5d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.340586 738 tablet_service.cc:1467] Processing CreateTablet for tablet 46cdb418ef204387939ac72a8bfc06ab (DEFAULT_TABLE table=default.Table_59bfceefa0364022aa60152f99bcff6f [id=dd42e2b4ae474dcc93718276325f5d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.340772 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cdb418ef204387939ac72a8bfc06ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.340971 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cdb418ef204387939ac72a8bfc06ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.342573 1178 ts_tablet_manager.cc:1428] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 20:51:53.342826 1178 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.344135 1178 tablet_bootstrap.cc:654] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.345817 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 8836c6c1460f41cc8d52d328828e68f9 (DEFAULT_TABLE table=default.Table_9647534e6db94ff79224863b3f8ae645 [id=8b986eae27b9454b9f8e344907af123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.346199 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8836c6c1460f41cc8d52d328828e68f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.346518 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8836c6c1460f41cc8d52d328828e68f9 (DEFAULT_TABLE table=default.Table_9647534e6db94ff79224863b3f8ae645 [id=8b986eae27b9454b9f8e344907af123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.346858 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8836c6c1460f41cc8d52d328828e68f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.347404 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1041) took 3426 ms. Trace:
I20250112 20:51:53.347530 2447 rpcz_store.cc:276] 0112 20:51:49.920568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.920610 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:53.336254 (+3415644us) ts_tablet_manager.cc:1145] Deleting tablet c90c8dd4dfb14507accb82828be512f0
0112 20:51:53.336264 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.347381 (+ 11117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342687}
I20250112 20:51:53.348145 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1038) took 3428 ms. Trace:
I20250112 20:51:53.348309 2447 rpcz_store.cc:276] 0112 20:51:49.919441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.920502 (+ 1061us) service_pool.cc:225] Handling call
0112 20:51:53.347949 (+3427447us) ts_tablet_manager.cc:1145] Deleting tablet ced81345e31a4634bb21e3702ad69c93
0112 20:51:53.347984 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.348120 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3354389}
I20250112 20:51:53.349586 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1039) took 3429 ms. Trace:
I20250112 20:51:53.350111 2447 rpcz_store.cc:276] 0112 20:51:49.920379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.037749 (+117370us) service_pool.cc:225] Handling call
0112 20:51:53.348712 (+3310963us) ts_tablet_manager.cc:1145] Deleting tablet 7623bebb539e485aa508e95a35225979
0112 20:51:53.348722 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.349465 (+ 743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3310737}
I20250112 20:51:53.353088 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1053) took 3208 ms. Trace:
I20250112 20:51:53.345817 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2d80d637134523a2e5796d4471ef9c (DEFAULT_TABLE table=default.Table_cfebf90ca7954edfbdc4cec024e66dd0 [id=d803f89acdda4b00ab5e8bf27f83a509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.354079 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2d80d637134523a2e5796d4471ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.354367 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 15a3be119a7741ba804ed454756bdeee (DEFAULT_TABLE table=default.Table_631ecd1223de417bbc68cb50d4805337 [id=eefe336440324493871e93447300f2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.354683 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a3be119a7741ba804ed454756bdeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.356895 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 601e1ce671f343fab954c7b8811d1520 (DEFAULT_TABLE table=default.Table_a29c589a2f7043e1939ce20fa09bc1f3 [id=5aa934e20e6d470ab0be9b9c5c157150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.357236 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601e1ce671f343fab954c7b8811d1520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.357884 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 (DEFAULT_TABLE table=default.Table_e553dd8387024dbca022b562a69871a5 [id=5d34e35ac96147aea870ed0c61744665]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.358188 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8fe3004a234c83a4dc76b9ed815d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.358654 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 (DEFAULT_TABLE table=default.Table_e553dd8387024dbca022b562a69871a5 [id=5d34e35ac96147aea870ed0c61744665]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.359826 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b370f96044305a79ebdb79d5e01d1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:53.360663 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 68866ef888d6413597b28e4813b92fc4 (DEFAULT_TABLE table=default.Table_9065cbdefe5349b5b6c70898bec1b850 [id=d4a77fee03a548658d9c0488e387704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.360834 899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e2c3f72ab4f439678d166b9b31661" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:53.361029 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68866ef888d6413597b28e4813b92fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.361057 899 raft_consensus.cc:3054] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.363581 899 raft_consensus.cc:2463] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:53.363556 3438 tablet.cc:1721] T 61fac2cf43c24c158e340cbc8dd176c3 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.363837 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263b370f96044305a79ebdb79d5e01d1" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:53.364003 911 raft_consensus.cc:3054] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.365672 745 tablet_service.cc:1467] Processing CreateTablet for tablet 68866ef888d6413597b28e4813b92fc4 (DEFAULT_TABLE table=default.Table_9065cbdefe5349b5b6c70898bec1b850 [id=d4a77fee03a548658d9c0488e387704c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:53.366273 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet c90c8dd4dfb14507accb82828be512f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c90c8dd4dfb14507accb82828be512f0 already in progress: creating tablet
W20250112 20:51:53.366470 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet ced81345e31a4634bb21e3702ad69c93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced81345e31a4634bb21e3702ad69c93 already in progress: creating tablet
I20250112 20:51:53.366470 911 raft_consensus.cc:2463] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
W20250112 20:51:53.366632 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 7623bebb539e485aa508e95a35225979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7623bebb539e485aa508e95a35225979 already in progress: creating tablet
I20250112 20:51:53.367470 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc5911185744cbbc028dcbc85f51d6 (DEFAULT_TABLE table=default.Table_ccc15f3eb2cd41aeac09ecb6e04fa79f [id=823db6804a984d14a40a938d3de95936]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.367787 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edc5911185744cbbc028dcbc85f51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.369132 751 tablet_service.cc:1467] Processing CreateTablet for tablet 15a3be119a7741ba804ed454756bdeee (DEFAULT_TABLE table=default.Table_631ecd1223de417bbc68cb50d4805337 [id=eefe336440324493871e93447300f2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.369561 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e2c3f72ab4f439678d166b9b31661" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:53.370149 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8fe3004a234c83a4dc76b9ed815d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.370605 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68866ef888d6413597b28e4813b92fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.381186 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a3be119a7741ba804ed454756bdeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.384310 710 leader_election.cc:304] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.384701 710 leader_election.cc:304] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.384950 3392 raft_consensus.cc:2798] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:53.384989 710 leader_election.cc:343] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:53.385057 3392 raft_consensus.cc:695] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
W20250112 20:51:53.385162 710 consensus_peers.cc:487] T c7a57ea399194a45bc8990a029550440 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7a57ea399194a45bc8990a029550440. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.385272 3392 consensus_queue.cc:237] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.384742 3121 raft_consensus.cc:2798] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.386855 738 tablet_service.cc:1467] Processing CreateTablet for tablet 601e1ce671f343fab954c7b8811d1520 (DEFAULT_TABLE table=default.Table_a29c589a2f7043e1939ce20fa09bc1f3 [id=5aa934e20e6d470ab0be9b9c5c157150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.386934 3121 raft_consensus.cc:695] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:53.387180 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601e1ce671f343fab954c7b8811d1520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.387148 3121 consensus_queue.cc:237] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.390156 748 tablet_service.cc:1467] Processing CreateTablet for tablet ade3a33dfabb42499213ada871ddb7d8 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.390528 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade3a33dfabb42499213ada871ddb7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.390769 745 tablet_service.cc:1467] Processing CreateTablet for tablet f9aeeae000954da6a198e9b99520421a (DEFAULT_TABLE table=default.Table_7ff1c2ec5f8d4ed5ad981b81dfa6cd73 [id=948efa3e278347d99201ed36857d380b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.390858 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea59f9de2224fb095fa92222bc47c6f (DEFAULT_TABLE table=default.Table_e929ae2e87394e6fbec49a339af2de71 [id=ac3f5fc9e3874bd3bb9564fc7c1c58bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.391091 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9aeeae000954da6a198e9b99520421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.391201 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ade3a33dfabb42499213ada871ddb7d8 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.391911 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea59f9de2224fb095fa92222bc47c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.392199 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade3a33dfabb42499213ada871ddb7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.392553 739 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc5911185744cbbc028dcbc85f51d6 (DEFAULT_TABLE table=default.Table_ccc15f3eb2cd41aeac09ecb6e04fa79f [id=823db6804a984d14a40a938d3de95936]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.392935 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edc5911185744cbbc028dcbc85f51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.401083 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2d80d637134523a2e5796d4471ef9c (DEFAULT_TABLE table=default.Table_cfebf90ca7954edfbdc4cec024e66dd0 [id=d803f89acdda4b00ab5e8bf27f83a509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.401541 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2d80d637134523a2e5796d4471ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.413400 1164 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:53.413554 1164 ts_tablet_manager.cc:1397] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.282s user 0.010s sys 0.002s
I20250112 20:51:53.415711 1022 tablet_service.cc:1467] Processing CreateTablet for tablet f9aeeae000954da6a198e9b99520421a (DEFAULT_TABLE table=default.Table_7ff1c2ec5f8d4ed5ad981b81dfa6cd73 [id=948efa3e278347d99201ed36857d380b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.416105 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9aeeae000954da6a198e9b99520421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.417717 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8582f6d6b1204d42b10e3c0e7cea489a (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.418059 756 tablet_service.cc:1467] Processing CreateTablet for tablet 81d195c999054b42b86f7fa39a628327 (DEFAULT_TABLE table=default.Table_b2c5c4da479a4eef84e80dfba42bfc8c [id=64cee882f9b840eda01a4d8a837cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.418144 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8582f6d6b1204d42b10e3c0e7cea489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.418403 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d195c999054b42b86f7fa39a628327. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.420486 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 81d195c999054b42b86f7fa39a628327 (DEFAULT_TABLE table=default.Table_b2c5c4da479a4eef84e80dfba42bfc8c [id=64cee882f9b840eda01a4d8a837cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.420996 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d195c999054b42b86f7fa39a628327. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:53.422516 983 consensus_peers.cc:487] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24a5162a25ce47519a091742f3d5ca31. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:53.424712 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffe14c8f0406451bae99014c98c6cae1 consensus state has no leader
I20250112 20:51:53.426483 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:53.426934 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:53.427254 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:53.427659 455 catalog_manager.cc:5526] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "87ae45957c64469d96dc92d606b5b4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:53.428423 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 263b370f96044305a79ebdb79d5e01d1 (table default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4]). Expected version 1 got 0
I20250112 20:51:53.429375 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 8582f6d6b1204d42b10e3c0e7cea489a (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.429925 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8582f6d6b1204d42b10e3c0e7cea489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.439404 1163 tablet_bootstrap.cc:492] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.439603 1163 ts_tablet_manager.cc:1397] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.002s
I20250112 20:51:53.440508 747 tablet_service.cc:1467] Processing CreateTablet for tablet b071742ab5fc490aa4d53ba4edaa5d63 (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.440920 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b071742ab5fc490aa4d53ba4edaa5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.441388 756 tablet_service.cc:1467] Processing CreateTablet for tablet b8a5099cd176493ba4e5c89c648ae55f (DEFAULT_TABLE table=default.Table_3c90a3a974a141558db73a01cfa97722 [id=9ca04190f5124f7e9220e0fc7fd7a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.441910 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a5099cd176493ba4e5c89c648ae55f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:53.442224 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f84396f2bf414bac8fc03664e0cb4579 consensus state has no leader
I20250112 20:51:53.443876 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6a54f4c3294580bf205364384eb324 (DEFAULT_TABLE table=default.Table_fe4fe28686e24c5c8505eeda33fe9440 [id=8b061c4f8a114b4692915c21643d9819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.444275 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6a54f4c3294580bf205364384eb324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.445029 737 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6a54f4c3294580bf205364384eb324 (DEFAULT_TABLE table=default.Table_fe4fe28686e24c5c8505eeda33fe9440 [id=8b061c4f8a114b4692915c21643d9819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.445351 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6a54f4c3294580bf205364384eb324. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:53.445669 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 59894.6us
W20250112 20:51:53.445871 710 leader_election.cc:343] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:53.446220 710 leader_election.cc:343] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:53.446646 710 leader_election.cc:343] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:53.446936 710 leader_election.cc:343] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:53.447026 710 leader_election.cc:304] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.447239 3114 raft_consensus.cc:2743] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 20:51:53.447530 710 leader_election.cc:343] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:53.447487 3114 consensus_queue.cc:1035] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.450870 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:52 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.451081 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b67d66724e247039cef64b54c4aef07) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:53.452437 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aeb83d80e48b4f4081920ffb0f121f2e consensus state has no leader
I20250112 20:51:53.455937 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_945a9b6afcd24ea0a35e62e93fba3d5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.456439 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945a9b6afcd24ea0a35e62e93fba3d5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.457562 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_088aef221fd440b1a66db4f07efa4232" } modify_external_catalogs: true
I20250112 20:51:53.457707 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_088aef221fd440b1a66db4f07efa4232" } modify_external_catalogs: true
I20250112 20:51:53.458513 1023 tablet_service.cc:1467] Processing CreateTablet for tablet b071742ab5fc490aa4d53ba4edaa5d63 (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.458884 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b071742ab5fc490aa4d53ba4edaa5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.353581 2447 rpcz_store.cc:276] 0112 20:51:50.144365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.176653 (+ 32288us) service_pool.cc:225] Handling call
0112 20:51:53.351873 (+3175220us) ts_tablet_manager.cc:1145] Deleting tablet 8bc545822e98460e815df4e6cad455b4
0112 20:51:53.351998 (+ 125us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.352931 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3173740}
I20250112 20:51:53.465538 3426 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:53.466538 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea59f9de2224fb095fa92222bc47c6f (DEFAULT_TABLE table=default.Table_e929ae2e87394e6fbec49a339af2de71 [id=ac3f5fc9e3874bd3bb9564fc7c1c58bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:53.467607 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8bc545822e98460e815df4e6cad455b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc545822e98460e815df4e6cad455b4 already in progress: creating tablet
I20250112 20:51:53.460935 1013 tablet_service.cc:1467] Processing CreateTablet for tablet b8a5099cd176493ba4e5c89c648ae55f (DEFAULT_TABLE table=default.Table_3c90a3a974a141558db73a01cfa97722 [id=9ca04190f5124f7e9220e0fc7fd7a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.468016 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.414229 1164 raft_consensus.cc:357] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.467904 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1058) took 3286 ms. Trace:
I20250112 20:51:53.468320 1164 raft_consensus.cc:383] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.468809 888 tablet_service.cc:1467] Processing CreateTablet for tablet abd40e0cf7ae446182c498e3dedc69ec (DEFAULT_TABLE table=default.Table_a0aadee20eb34f1eb4f122d7d5920824 [id=953c59739f10479eb4960753db269c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.469023 1164 raft_consensus.cc:738] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.469458 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd40e0cf7ae446182c498e3dedc69ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.468727 2447 rpcz_store.cc:276] 0112 20:51:50.180834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.180887 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:53.467124 (+3286237us) ts_tablet_manager.cc:1145] Deleting tablet 5065a94bb13c419e9d9b8eaca72f5052
0112 20:51:53.467143 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.467708 (+ 565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3286063}
I20250112 20:51:53.471081 879 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2d80d637134523a2e5796d4471ef9c (DEFAULT_TABLE table=default.Table_cfebf90ca7954edfbdc4cec024e66dd0 [id=d803f89acdda4b00ab5e8bf27f83a509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.471392 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2d80d637134523a2e5796d4471ef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.471186 1163 raft_consensus.cc:357] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.475936 448 master_service.cc:946] ReplaceTablet: received request to replace tablet c90c8dd4dfb14507accb82828be512f0 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:53.476083 889 tablet_service.cc:1467] Processing CreateTablet for tablet 46cdb418ef204387939ac72a8bfc06ab (DEFAULT_TABLE table=default.Table_59bfceefa0364022aa60152f99bcff6f [id=dd42e2b4ae474dcc93718276325f5d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.476387 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cdb418ef204387939ac72a8bfc06ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.479265 1163 raft_consensus.cc:383] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.479341 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 65784493c6ef417993b1f6371cab47d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.479403 1163 raft_consensus.cc:738] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.479619 1163 consensus_queue.cc:260] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.470535 1164 consensus_queue.cc:260] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.481964 1164 ts_tablet_manager.cc:1428] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20250112 20:51:53.482262 1164 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:53.483686 3258 log.cc:926] Time spent T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20250112 20:51:53.484191 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8836c6c1460f41cc8d52d328828e68f9 (DEFAULT_TABLE table=default.Table_9647534e6db94ff79224863b3f8ae645 [id=8b986eae27b9454b9f8e344907af123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.484741 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8836c6c1460f41cc8d52d328828e68f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.488927 3426 tablet.cc:1721] T db4fa71cf46144629e1fc1f2172fc3cb P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.490345 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea59f9de2224fb095fa92222bc47c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.490234 3427 tablet.cc:1721] T 36935e6d8c5643388ad163458e2b387c P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.492728 1047 raft_consensus.cc:3054] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:53.496925 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 5065a94bb13c419e9d9b8eaca72f5052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5065a94bb13c419e9d9b8eaca72f5052 already in progress: creating tablet
I20250112 20:51:53.497032 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77" } modify_external_catalogs: true
I20250112 20:51:53.497200 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_03f479f5b1144f1cbc7bbbe3b7f9ae77" } modify_external_catalogs: true
I20250112 20:51:53.497318 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3d064f1656bd4dfc83bf03f62daea2d3" } modify_external_catalogs: true
I20250112 20:51:53.497452 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3d064f1656bd4dfc83bf03f62daea2d3" } modify_external_catalogs: true
I20250112 20:51:53.497574 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3ac63da2597b4f6a87e1619e911a5ee7" } modify_external_catalogs: true
I20250112 20:51:53.497620 888 tablet_service.cc:1467] Processing CreateTablet for tablet 601e1ce671f343fab954c7b8811d1520 (DEFAULT_TABLE table=default.Table_a29c589a2f7043e1939ce20fa09bc1f3 [id=5aa934e20e6d470ab0be9b9c5c157150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.497701 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3ac63da2597b4f6a87e1619e911a5ee7" } modify_external_catalogs: true
I20250112 20:51:53.497995 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 601e1ce671f343fab954c7b8811d1520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.503351 884 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8fe3004a234c83a4dc76b9ed815d30 (DEFAULT_TABLE table=default.Table_e553dd8387024dbca022b562a69871a5 [id=5d34e35ac96147aea870ed0c61744665]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.503679 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8fe3004a234c83a4dc76b9ed815d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.504302 879 tablet_service.cc:1467] Processing CreateTablet for tablet 15a3be119a7741ba804ed454756bdeee (DEFAULT_TABLE table=default.Table_631ecd1223de417bbc68cb50d4805337 [id=eefe336440324493871e93447300f2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.504717 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a3be119a7741ba804ed454756bdeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.509455 3438 tablet.cc:1721] T db4fa71cf46144629e1fc1f2172fc3cb P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.512905 884 tablet_service.cc:1467] Processing CreateTablet for tablet 68866ef888d6413597b28e4813b92fc4 (DEFAULT_TABLE table=default.Table_9065cbdefe5349b5b6c70898bec1b850 [id=d4a77fee03a548658d9c0488e387704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.513572 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68866ef888d6413597b28e4813b92fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.517274 891 tablet_service.cc:1467] Processing CreateTablet for tablet f9aeeae000954da6a198e9b99520421a (DEFAULT_TABLE table=default.Table_7ff1c2ec5f8d4ed5ad981b81dfa6cd73 [id=948efa3e278347d99201ed36857d380b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.517568 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9aeeae000954da6a198e9b99520421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.518312 3427 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:53.519369 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc5911185744cbbc028dcbc85f51d6 (DEFAULT_TABLE table=default.Table_ccc15f3eb2cd41aeac09ecb6e04fa79f [id=823db6804a984d14a40a938d3de95936]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.519640 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edc5911185744cbbc028dcbc85f51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.522435 3427 tablet.cc:1721] T 61fac2cf43c24c158e340cbc8dd176c3 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.523269 1163 ts_tablet_manager.cc:1428] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.084s user 0.001s sys 0.001s
I20250112 20:51:53.523365 885 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea59f9de2224fb095fa92222bc47c6f (DEFAULT_TABLE table=default.Table_e929ae2e87394e6fbec49a339af2de71 [id=ac3f5fc9e3874bd3bb9564fc7c1c58bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.523787 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea59f9de2224fb095fa92222bc47c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.531370 891 tablet_service.cc:1467] Processing CreateTablet for tablet ade3a33dfabb42499213ada871ddb7d8 (DEFAULT_TABLE table=default.Table_f8b3eae7e0294abfa7f8f431ff6aee97 [id=e3236c5610634f9eabb22f11d01ac4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.531730 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade3a33dfabb42499213ada871ddb7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.535188 3426 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.536185 3438 tablet.cc:1721] T 549d92191a594788bf639cd60a6b5142 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 20:51:53.537719 1163 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.541750 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6a54f4c3294580bf205364384eb324 (DEFAULT_TABLE table=default.Table_fe4fe28686e24c5c8505eeda33fe9440 [id=8b061c4f8a114b4692915c21643d9819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.542122 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6a54f4c3294580bf205364384eb324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.542624 882 tablet_service.cc:1467] Processing CreateTablet for tablet b071742ab5fc490aa4d53ba4edaa5d63 (DEFAULT_TABLE table=default.Table_79e838bb6ced422285e3bb8e26367f4f [id=18f1e59c09484bb498f5e993ed17c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.542908 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b071742ab5fc490aa4d53ba4edaa5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.543169 3438 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.544098 1185 ts_tablet_manager.cc:1918] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.547776 885 tablet_service.cc:1467] Processing CreateTablet for tablet b8a5099cd176493ba4e5c89c648ae55f (DEFAULT_TABLE table=default.Table_3c90a3a974a141558db73a01cfa97722 [id=9ca04190f5124f7e9220e0fc7fd7a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.548194 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a5099cd176493ba4e5c89c648ae55f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:53.550086 710 consensus_peers.cc:487] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db9bab2477a94c169817d9de8aaad1c0. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.554670 1185 log.cc:1198] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:53.553690 1164 tablet_bootstrap.cc:654] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.558522 1163 tablet_bootstrap.cc:654] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.559235 1185 ts_tablet_manager.cc:1939] T b85b9e82f9e847c4acfeecb82201e694 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:53.560943 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1093) took 3664 ms. Trace:
I20250112 20:51:53.561030 1185 rpcz_store.cc:276] 0112 20:51:49.896206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.913364 (+ 17158us) service_pool.cc:225] Handling call
0112 20:51:53.303994 (+3390630us) ts_tablet_manager.cc:1145] Deleting tablet b85b9e82f9e847c4acfeecb82201e694
0112 20:51:53.304007 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.331540 (+ 27533us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.544000 (+212460us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.560926 (+ 16926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390190}
I20250112 20:51:53.564560 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b85b9e82f9e847c4acfeecb82201e694 (table default.Table_7927297ff7264be5a6391d2a58f38280 [id=8530d1e179394e18ad4464ae2187dcf9]) successfully deleted
W20250112 20:51:53.564839 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 940756f8867b4cde91fb0e89e6069265 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940756f8867b4cde91fb0e89e6069265 already in progress: creating tablet
W20250112 20:51:53.565250 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82f5ebfcd8be4bac8cfb3e1689bf59dd already in progress: creating tablet
W20250112 20:51:53.565343 710 consensus_peers.cc:487] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf66492d01a64eb1b842acc279b8ee09. This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:53.565430 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73adfbbe4f8e4d08acfa97c6d9d69ce8 already in progress: creating tablet
W20250112 20:51:53.565611 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d396dfcbb5d04b82a52d4eda0ceb5ff2 already in progress: creating tablet
I20250112 20:51:53.567368 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8582f6d6b1204d42b10e3c0e7cea489a (DEFAULT_TABLE table=default.Table_54fe2ed58a8640d3abeb4b662e75a753 [id=e3b1101199f744acb80999d40ae16161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.567839 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8582f6d6b1204d42b10e3c0e7cea489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.570869 891 tablet_service.cc:1467] Processing CreateTablet for tablet 81d195c999054b42b86f7fa39a628327 (DEFAULT_TABLE table=default.Table_b2c5c4da479a4eef84e80dfba42bfc8c [id=64cee882f9b840eda01a4d8a837cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:53.571497 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d195c999054b42b86f7fa39a628327. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.574988 1164 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
W20250112 20:51:53.577670 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 54065.7us
I20250112 20:51:53.578785 3130 consensus_queue.cc:1035] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.579567 3130 consensus_queue.cc:1035] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.581992 901 raft_consensus.cc:3054] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.582507 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a5099cd176493ba4e5c89c648ae55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.586629 3130 consensus_queue.cc:1035] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.587725 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1060) took 3295 ms. Trace:
I20250112 20:51:53.587852 2447 rpcz_store.cc:276] 0112 20:51:50.292224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.297310 (+ 5086us) service_pool.cc:225] Handling call
0112 20:51:53.471008 (+3173698us) ts_tablet_manager.cc:1145] Deleting tablet 6181b166b5cb429da3ebba23cf267942
0112 20:51:53.587554 (+116546us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.587688 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3150459}
I20250112 20:51:53.588235 2447 tablet_replica.cc:331] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:53.588467 2447 raft_consensus.cc:2238] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:53.590966 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 6181b166b5cb429da3ebba23cf267942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6181b166b5cb429da3ebba23cf267942 already in progress: creating tablet
I20250112 20:51:53.593297 2447 raft_consensus.cc:2267] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.594491 2447 ts_tablet_manager.cc:1905] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.597136 3427 tablet.cc:1721] T db4fa71cf46144629e1fc1f2172fc3cb P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:53.604664 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_568b6570775647ed804c53da8224a41c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.607772 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_568b6570775647ed804c53da8224a41c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.611169 1163 tablet_bootstrap.cc:492] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.611367 1163 ts_tablet_manager.cc:1397] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.000s
I20250112 20:51:53.612212 1163 raft_consensus.cc:357] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.613409 1163 raft_consensus.cc:383] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.613641 1163 raft_consensus.cc:738] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.613955 1163 consensus_queue.cc:260] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:53.615231 1163 ts_tablet_manager.cc:1428] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250112 20:51:53.619479 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9206fc7b077452eb90acb15a50e71a4 consensus state has no leader
W20250112 20:51:53.629171 983 consensus_peers.cc:487] T 725e56fa8ac343e5b1ec1214da02b7d4 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 725e56fa8ac343e5b1ec1214da02b7d4. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.629639 1164 ts_tablet_manager.cc:1397] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.147s user 0.002s sys 0.002s
I20250112 20:51:53.630386 1164 raft_consensus.cc:357] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.630582 1164 raft_consensus.cc:383] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.630645 1164 raft_consensus.cc:738] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.630859 1164 consensus_queue.cc:260] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.631405 1164 ts_tablet_manager.cc:1428] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:53.631675 1164 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:53.628991 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1100) took 3728 ms. Trace:
I20250112 20:51:53.633247 1185 rpcz_store.cc:276] 0112 20:51:49.900510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.913660 (+ 13150us) service_pool.cc:225] Handling call
0112 20:51:53.628873 (+3715213us) ts_tablet_manager.cc:1145] Deleting tablet 8c5b10576bd34af5b02f8d8dcc3759ab
0112 20:51:53.628884 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.628967 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3715085}
W20250112 20:51:53.633720 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8c5b10576bd34af5b02f8d8dcc3759ab because the tablet was not found. No further retry: Not found: Tablet not found: 8c5b10576bd34af5b02f8d8dcc3759ab
I20250112 20:51:53.634527 1185 tablet_replica.cc:331] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:53.635092 1178 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.635233 1178 ts_tablet_manager.cc:1397] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.292s user 0.015s sys 0.027s
I20250112 20:51:53.635912 1178 raft_consensus.cc:357] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.636096 1178 raft_consensus.cc:383] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.636139 1178 raft_consensus.cc:738] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.636330 1178 consensus_queue.cc:260] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.636950 1178 ts_tablet_manager.cc:1428] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:53.637382 1178 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.638844 1178 tablet_bootstrap.cc:654] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.641251 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_acdb4e459bbd4e91bc58571fee9062f3" } modify_external_catalogs: true
I20250112 20:51:53.641472 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_acdb4e459bbd4e91bc58571fee9062f3" } modify_external_catalogs: true
I20250112 20:51:53.641502 1178 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.641713 1178 ts_tablet_manager.cc:1397] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250112 20:51:53.642274 1185 raft_consensus.cc:2238] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.642418 1185 pending_rounds.cc:62] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Trying to abort 1 pending ops.
I20250112 20:51:53.642642 1178 raft_consensus.cc:357] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.642949 1178 raft_consensus.cc:383] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.643064 1178 raft_consensus.cc:738] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.643343 1178 consensus_queue.cc:260] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.644205 1178 ts_tablet_manager.cc:1428] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:53.644580 1178 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.646690 1178 tablet_bootstrap.cc:654] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.647444 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_ba62c67a244a4f66b09888d3b055c85b [id=91351703475c45e69c55a3ee47c4bb51] alter complete (version 2)
I20250112 20:51:53.660142 1163 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.660830 3485 tablet.cc:1721] T c79ce3118cbf49e287c93a76398521da P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 20:51:53.661900 1163 tablet_bootstrap.cc:654] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.670837 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet db4fa71cf46144629e1fc1f2172fc3cb (table default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513]). Expected version 1 got 0
I20250112 20:51:53.671146 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.671423 1186 tablet_replica.cc:331] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:53.671561 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.671391 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.642458 1185 pending_rounds.cc:69] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585088551563264 op_type: NO_OP noop_request { }
I20250112 20:51:53.673835 1185 raft_consensus.cc:2883] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:53.673995 1185 raft_consensus.cc:2267] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.674569 1185 ts_tablet_manager.cc:1905] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.687633 1164 tablet_bootstrap.cc:654] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.688541 1163 tablet_bootstrap.cc:492] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.688795 1163 ts_tablet_manager.cc:1397] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20250112 20:51:53.689741 1163 raft_consensus.cc:357] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.690038 1163 raft_consensus.cc:383] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.690160 1163 raft_consensus.cc:738] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.690474 1163 consensus_queue.cc:260] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.691464 1163 ts_tablet_manager.cc:1428] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:53.691814 1163 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.693583 1163 tablet_bootstrap.cc:654] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.693778 1186 raft_consensus.cc:2238] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.694108 1186 pending_rounds.cc:62] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Trying to abort 1 pending ops.
I20250112 20:51:53.694162 1186 pending_rounds.cc:69] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585101061271552 op_type: NO_OP noop_request { }
I20250112 20:51:53.694262 1186 raft_consensus.cc:2883] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:53.694347 1186 raft_consensus.cc:2267] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.696430 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_83f956b78be94785963dbdbbe1fb3016"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.697347 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83f956b78be94785963dbdbbe1fb3016 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.698726 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_5252a0d0e4bc443cae9a85749493f7b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.699295 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5252a0d0e4bc443cae9a85749493f7b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.700407 1186 ts_tablet_manager.cc:1905] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.707471 1186 ts_tablet_manager.cc:1918] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.707623 1186 log.cc:1198] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/8dafaa3a1a3d4d8bb8aa0448a61e237c
I20250112 20:51:53.702273 3476 consensus_queue.cc:1035] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.708202 1186 ts_tablet_manager.cc:1939] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:53.709619 1186 tablet_replica.cc:331] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:53.709700 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c (table default.Table_314fc7eb6aff43d7b18dba3be1a7c618 [id=4509ee4f763f4fcea8b6eca124310e7a]) successfully deleted
I20250112 20:51:53.709934 1186 raft_consensus.cc:2238] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.710147 1186 raft_consensus.cc:2267] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.710855 1186 ts_tablet_manager.cc:1905] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:53.711371 983 consensus_peers.cc:487] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.715327 2447 ts_tablet_manager.cc:1918] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.715489 2447 log.cc:1198] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/61b74aaf7f5b4c87acbefaa4df5c10d2
I20250112 20:51:53.715898 2447 ts_tablet_manager.cc:1939] T 61b74aaf7f5b4c87acbefaa4df5c10d2 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:53.716279 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1062) took 3414 ms. Trace:
I20250112 20:51:53.721161 1163 tablet_bootstrap.cc:492] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.721377 1163 ts_tablet_manager.cc:1397] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20250112 20:51:53.722502 1163 raft_consensus.cc:357] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.722805 1163 raft_consensus.cc:383] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.722934 1163 raft_consensus.cc:738] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.723246 1163 consensus_queue.cc:260] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.723937 1163 ts_tablet_manager.cc:1428] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:53.724298 1163 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.727013 1163 tablet_bootstrap.cc:654] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.729983 447 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:53.732210 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.736812 1163 tablet_bootstrap.cc:492] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.737260 1163 ts_tablet_manager.cc:1397] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250112 20:51:53.739895 1163 raft_consensus.cc:357] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.740912 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet e9202294cd5d4aa9b5396ebdd65abda3
I20250112 20:51:53.741101 1163 raft_consensus.cc:383] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.741494 1163 raft_consensus.cc:738] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.743044 1163 consensus_queue.cc:260] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.746368 1163 ts_tablet_manager.cc:1428] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250112 20:51:53.747788 1163 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.748534 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:53.749608 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:53.751956 1186 ts_tablet_manager.cc:1918] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.752651 1186 log.cc:1198] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/b30a34c1abcb4f72a7f4f8a8f5be3720
I20250112 20:51:53.754846 1186 ts_tablet_manager.cc:1939] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:53.756368 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 86e14004e8c546d38a38add58c9a243f
I20250112 20:51:53.758045 1163 tablet_bootstrap.cc:654] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.759408 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.759459 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_cffbc7933e2945b1b3dd4e9a4a48ef8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.759831 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet b30a34c1abcb4f72a7f4f8a8f5be3720 (table default.Table_9ff8fdbb27a140239a8286434feff5a2 [id=d518a4c7edde4ba3b05d700c1cdb1dd1]) successfully deleted
W20250112 20:51:53.759971 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cffbc7933e2945b1b3dd4e9a4a48ef8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.760262 1186 tablet_replica.cc:331] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:53.761252 1186 raft_consensus.cc:2238] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.761904 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.762219 1186 raft_consensus.cc:2267] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.765162 1186 ts_tablet_manager.cc:1905] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.765960 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b1edc79f17f4bb7a6bb9ab619403142
I20250112 20:51:53.767479 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:57104
W20250112 20:51:53.767717 3258 log.cc:926] Time spent T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250112 20:51:53.772478 1163 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.773005 1163 ts_tablet_manager.cc:1397] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20250112 20:51:53.777555 1163 raft_consensus.cc:357] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.779253 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.780180 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e8afee11cf87476ba2f54ed148016505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.780711 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8afee11cf87476ba2f54ed148016505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.779479 1163 raft_consensus.cc:383] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.782025 1163 raft_consensus.cc:738] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.782418 1163 consensus_queue.cc:260] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.783336 1163 ts_tablet_manager.cc:1428] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250112 20:51:53.785260 1163 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:53.741953 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e9202294cd5d4aa9b5396ebdd65abda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.767287 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1edc79f17f4bb7a6bb9ab619403142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.777343 3392 consensus_queue.cc:1035] T db9bab2477a94c169817d9de8aaad1c0 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.790117 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_2b7c53013efb48b993883f116f6cb05c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:53.790656 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b7c53013efb48b993883f116f6cb05c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.794399 1163 tablet_bootstrap.cc:654] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.794865 3476 raft_consensus.cc:491] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.794999 3476 raft_consensus.cc:513] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.795629 3476 leader_election.cc:290] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.795917 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet f7f491d69a704f32899e231503da02d0
I20250112 20:51:53.796393 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008d69d199d64bdf85da17d8d3bf6264" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:53.796582 760 raft_consensus.cc:2463] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:53.716414 2447 rpcz_store.cc:276] 0112 20:51:50.301741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.301821 (+ 80us) service_pool.cc:225] Handling call
0112 20:51:53.588187 (+3286366us) ts_tablet_manager.cc:1145] Deleting tablet 61b74aaf7f5b4c87acbefaa4df5c10d2
0112 20:51:53.588196 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.703673 (+115477us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.715241 (+ 11568us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.716262 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4605,"tablet-delete.queue_time_us":3267569}
I20250112 20:51:53.797627 2447 tablet_replica.cc:331] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:53.797823 2447 raft_consensus.cc:2238] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.797863 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 61b74aaf7f5b4c87acbefaa4df5c10d2 (table default.Table_ac72a7aedf80477ea4d02c7a2f150f28 [id=cabf46d4a77c4569a41b919f9c77cb7d]) successfully deleted
W20250112 20:51:53.798074 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 54152.6us
I20250112 20:51:53.798209 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f491d69a704f32899e231503da02d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.801237 983 leader_election.cc:304] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.800725 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008d69d199d64bdf85da17d8d3bf6264" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:53.802112 912 raft_consensus.cc:2463] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:53.803635 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f491d69a704f32899e231503da02d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:53.805096 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f491d69a704f32899e231503da02d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:53.807240 1186 ts_tablet_manager.cc:1918] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.807632 1186 log.cc:1198] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/95cbe4933f2b49ebbdb6eb7f0f8db088
I20250112 20:51:53.808372 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_beaa9d2f9e5444baafe1a7b27a028b39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:53.808825 1186 ts_tablet_manager.cc:1939] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
W20250112 20:51:53.808948 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beaa9d2f9e5444baafe1a7b27a028b39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.810741 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 (table default.Table_8c9a7a06fb384d4aa61879e57247d36e [id=ce81dff010874269954c90b958f031ee]) successfully deleted
I20250112 20:51:53.810889 1186 tablet_replica.cc:331] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
W20250112 20:51:53.811338 710 consensus_peers.cc:487] T 5ccb593413124921af94d0b855789b52 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ccb593413124921af94d0b855789b52. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:53.811655 1186 raft_consensus.cc:2238] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.812320 1186 raft_consensus.cc:2267] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.814291 1186 ts_tablet_manager.cc:1905] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.818154 3476 raft_consensus.cc:2798] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.818248 3476 raft_consensus.cc:491] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.818311 3476 raft_consensus.cc:3054] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.820569 2447 raft_consensus.cc:2267] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.821238 2447 ts_tablet_manager.cc:1905] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.821669 3476 raft_consensus.cc:513] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.822955 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008d69d199d64bdf85da17d8d3bf6264" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:53.823132 760 raft_consensus.cc:3054] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.823256 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008d69d199d64bdf85da17d8d3bf6264" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:53.823422 901 raft_consensus.cc:3054] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.825863 760 raft_consensus.cc:2463] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:53.826201 901 raft_consensus.cc:2463] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:53.822365 3476 leader_election.cc:290] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.826956 983 leader_election.cc:304] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:53.827437 3476 raft_consensus.cc:2798] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.827549 3476 raft_consensus.cc:695] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:53.827756 3476 consensus_queue.cc:237] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.830860 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_fae781a8a94d42c9a6ede1c013980e8c [id=4d672d282c90487b80b232150877549a] alter complete (version 1)
I20250112 20:51:53.832306 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.834961 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_06a839a7eee64f6889596bab4fe192d8" } new_table_name: "default.Table_e523fe7f20374991a4bbfd1a6e23e05a" modify_external_catalogs: true
I20250112 20:51:53.837839 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:53.841722 3121 consensus_queue.cc:1035] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.842555 1178 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.842783 1178 ts_tablet_manager.cc:1397] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.198s user 0.026s sys 0.031s
I20250112 20:51:53.844012 1178 raft_consensus.cc:357] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.844331 1178 raft_consensus.cc:383] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.844439 1178 raft_consensus.cc:738] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.844815 1178 consensus_queue.cc:260] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.845893 1178 ts_tablet_manager.cc:1428] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:53.846208 1186 ts_tablet_manager.cc:1918] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:53.846741 1186 log.cc:1198] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/691e2c3f72ab4f439678d166b9b31661
I20250112 20:51:53.846935 1178 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.847817 1186 ts_tablet_manager.cc:1939] T 691e2c3f72ab4f439678d166b9b31661 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:53.848493 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_f8b3eae7e0294abfa7f8f431ff6aee97" } new_table_name: "default.Table_2b480927fa114bf89a25989a1a97d8de" modify_external_catalogs: true
I20250112 20:51:53.851197 1178 tablet_bootstrap.cc:654] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.851769 1185 ts_tablet_manager.cc:1918] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.851882 1185 log.cc:1198] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/e8bea4d83ea644a0930a29f61e7a5fc8
I20250112 20:51:53.852005 3114 raft_consensus.cc:491] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.852150 3114 raft_consensus.cc:513] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.852308 1185 ts_tablet_manager.cc:1939] T e8bea4d83ea644a0930a29f61e7a5fc8 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:53.852747 3114 leader_election.cc:290] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.853253 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 691e2c3f72ab4f439678d166b9b31661 (table default.Table_8df177b5b9c84139b3ac6874e7c7fd1c [id=cb04bfadae3b451893a83d27479e49dc]) successfully deleted
W20250112 20:51:53.853504 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet e9202294cd5d4aa9b5396ebdd65abda3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9202294cd5d4aa9b5396ebdd65abda3 already in progress: creating tablet
W20250112 20:51:53.853672 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b1edc79f17f4bb7a6bb9ab619403142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b1edc79f17f4bb7a6bb9ab619403142 already in progress: creating tablet
W20250112 20:51:53.853840 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet f7f491d69a704f32899e231503da02d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7f491d69a704f32899e231503da02d0 already in progress: creating tablet
I20250112 20:51:53.855998 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cbe4933f2b49ebbdb6eb7f0f8db088" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.856215 1048 raft_consensus.cc:2463] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:53.857172 710 leader_election.cc:304] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
W20250112 20:51:53.857425 710 leader_election.cc:343] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 95cbe4933f2b49ebbdb6eb7f0f8db088
I20250112 20:51:53.858119 3121 raft_consensus.cc:2798] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.858196 3121 raft_consensus.cc:491] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.858263 3121 raft_consensus.cc:3054] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.860949 3265 raft_consensus.cc:491] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.861066 3265 raft_consensus.cc:513] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.861222 1178 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:53.861404 1178 ts_tablet_manager.cc:1397] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20250112 20:51:53.861585 3265 leader_election.cc:290] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.862485 1178 raft_consensus.cc:357] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.862797 1178 raft_consensus.cc:383] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.862875 1178 raft_consensus.cc:738] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.863180 1178 consensus_queue.cc:260] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.863716 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30a34c1abcb4f72a7f4f8a8f5be3720" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:53.863790 710 leader_election.cc:343] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b30a34c1abcb4f72a7f4f8a8f5be3720
I20250112 20:51:53.863902 1178 ts_tablet_manager.cc:1428] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 20:51:53.864367 710 leader_election.cc:343] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:53.864463 710 leader_election.cc:304] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.864934 3114 raft_consensus.cc:2743] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:53.866269 1178 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:53.869482 1178 tablet_bootstrap.cc:654] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.872051 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_5a3edf19ce64498bbf0f8518ad5c4f68" } new_table_name: "default.Table_ff0a7317912c4e05b31cac70b0d57115" modify_external_catalogs: true
I20250112 20:51:53.874527 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1095) took 3975 ms. Trace:
I20250112 20:51:53.874917 1185 rpcz_store.cc:276] 0112 20:51:49.898671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.914005 (+ 15334us) service_pool.cc:225] Handling call
0112 20:51:53.634481 (+3720476us) ts_tablet_manager.cc:1145] Deleting tablet e8bea4d83ea644a0930a29f61e7a5fc8
0112 20:51:53.634491 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.775020 (+140529us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.851662 (+ 76642us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.874508 (+ 22846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3720321}
I20250112 20:51:53.878371 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_bf01bbcaf340447098cdbe875519b5e9 [id=60b7060bed91463c89327b040c5c4513] alter complete (version 1)
I20250112 20:51:53.878667 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet e8bea4d83ea644a0930a29f61e7a5fc8 (table default.Table_92f355d4058f40b6b0979ff3f3f1af14 [id=2049ad7ffcee4affa56490ef29e949da]) successfully deleted
I20250112 20:51:53.878700 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1092) took 3987 ms. Trace:
I20250112 20:51:53.878780 1185 rpcz_store.cc:276] 0112 20:51:49.891528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.914899 (+ 23371us) service_pool.cc:225] Handling call
0112 20:51:53.878118 (+3963219us) ts_tablet_manager.cc:1145] Deleting tablet 961a72a9d8a643cea743f3f021bd5a03
0112 20:51:53.878561 (+ 443us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.878680 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3963009}
I20250112 20:51:53.880868 1048 raft_consensus.cc:3054] T a20d85a4be874b0b90c528a631a29aed P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.865839 3121 raft_consensus.cc:513] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.881520 1185 tablet_replica.cc:331] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
W20250112 20:51:53.881915 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 961a72a9d8a643cea743f3f021bd5a03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961a72a9d8a643cea743f3f021bd5a03 already in progress: creating tablet
I20250112 20:51:53.881963 1185 raft_consensus.cc:2238] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:53.882694 1185 raft_consensus.cc:2267] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.882627 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cbe4933f2b49ebbdb6eb7f0f8db088" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:53.882819 1044 raft_consensus.cc:3054] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.885543 1044 raft_consensus.cc:2463] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:53.885998 710 leader_election.cc:304] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:53.881723 3121 leader_election.cc:290] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.888317 3265 raft_consensus.cc:2798] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 20:51:53.888358 710 leader_election.cc:343] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 95cbe4933f2b49ebbdb6eb7f0f8db088
I20250112 20:51:53.888466 3265 raft_consensus.cc:695] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:53.888897 3265 consensus_queue.cc:237] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.893579 2447 ts_tablet_manager.cc:1918] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:53.889156 1185 ts_tablet_manager.cc:1905] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.896608 1164 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:53.896863 1164 ts_tablet_manager.cc:1397] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.265s user 0.002s sys 0.003s
I20250112 20:51:53.897843 1164 raft_consensus.cc:357] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.900038 1164 raft_consensus.cc:383] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.901078 2447 log.cc:1198] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/104d6b4effb84314a6b2b9db883b4366
I20250112 20:51:53.901849 2447 ts_tablet_manager.cc:1939] T 104d6b4effb84314a6b2b9db883b4366 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:53.905583 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1064) took 3602 ms. Trace:
I20250112 20:51:53.906184 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_af56f0d63dbc4b06894f55724dc835d6 [id=c7f63522332840b6bc6cd7462015ad3d] alter complete (version 2)
I20250112 20:51:53.911653 2447 rpcz_store.cc:276] 0112 20:51:50.302719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.302946 (+ 227us) service_pool.cc:225] Handling call
0112 20:51:53.797571 (+3494625us) ts_tablet_manager.cc:1145] Deleting tablet 104d6b4effb84314a6b2b9db883b4366
0112 20:51:53.797583 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.862259 (+ 64676us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.893468 (+ 31209us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.905564 (+ 12096us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3476624}
I20250112 20:51:53.913923 2447 tablet_replica.cc:331] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:53.915365 2447 raft_consensus.cc:2238] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.919284 3178 raft_consensus.cc:491] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.919430 3178 raft_consensus.cc:513] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.919982 3178 leader_election.cc:290] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:53.920145 2447 raft_consensus.cc:2267] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:53.921631 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ce9397b884b0095dae39ea57f5fd9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:53.922115 847 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:53.926462 2447 ts_tablet_manager.cc:1905] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:53.921659 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ce9397b884b0095dae39ea57f5fd9" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:53.931268 847 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:53.931335 1164 raft_consensus.cc:738] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.931381 847 leader_election.cc:304] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:53.931550 1164 consensus_queue.cc:260] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.931834 3178 raft_consensus.cc:2743] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:53.932125 1164 ts_tablet_manager.cc:1428] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250112 20:51:53.932385 1164 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:53.932125 3391 consensus_queue.cc:1035] T 07c83f45bd3d469ca55b3d9a5810d27d P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.913077 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 104d6b4effb84314a6b2b9db883b4366 (table default.Table_c456c3048d45444f9422bf7a190b290a [id=53f61c1b5450455cbc04eb4a76341ceb]) successfully deleted
I20250112 20:51:53.945588 3121 raft_consensus.cc:491] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.945801 3121 raft_consensus.cc:513] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.946579 3121 leader_election.cc:290] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:53.947268 1164 tablet_bootstrap.cc:654] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.947458 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d341f6ac6848d9a8e60f689dd24a66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:53.947451 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d341f6ac6848d9a8e60f689dd24a66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:53.947626 1047 raft_consensus.cc:2463] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:53.978983 3476 raft_consensus.cc:491] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.979235 3476 raft_consensus.cc:513] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.980542 3476 leader_election.cc:290] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:53.987180 983 leader_election.cc:343] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 615d75bc22df45cc876788a266e983cc
I20250112 20:51:53.989784 1163 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:53.989960 1163 ts_tablet_manager.cc:1397] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.205s user 0.011s sys 0.007s
I20250112 20:51:53.991084 1163 raft_consensus.cc:357] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.991369 1163 raft_consensus.cc:383] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.991470 1163 raft_consensus.cc:738] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
W20250112 20:51:53.948180 710 leader_election.cc:343] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:53.972204 1185 ts_tablet_manager.cc:1918] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:53.992319 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 44141.1us
I20250112 20:51:53.992359 1185 log.cc:1198] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f76dbd7f681b418ca7bb6c6e8f1f1e58
I20250112 20:51:53.992514 710 leader_election.cc:304] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:53.992847 1185 ts_tablet_manager.cc:1939] T f76dbd7f681b418ca7bb6c6e8f1f1e58 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:53.993420 760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:53.993669 760 raft_consensus.cc:2388] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30ba399c7a6749e0acd075d93e4cc4b3 in current term 1: Already voted for candidate 87ae45957c64469d96dc92d606b5b4b3 in this term.
I20250112 20:51:53.994161 3391 raft_consensus.cc:2798] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.994244 3391 raft_consensus.cc:491] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.994310 3391 raft_consensus.cc:3054] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.993675 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1096) took 4094 ms. Trace:
I20250112 20:51:53.991808 1163 consensus_queue.cc:260] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.998610 1185 rpcz_store.cc:276] 0112 20:51:49.898940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.915635 (+ 16695us) service_pool.cc:225] Handling call
0112 20:51:53.881472 (+3965837us) ts_tablet_manager.cc:1145] Deleting tablet f76dbd7f681b418ca7bb6c6e8f1f1e58
0112 20:51:53.881483 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.898762 (+ 17279us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.971917 (+ 73155us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:53.993653 (+ 21736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3965621}
I20250112 20:51:53.998937 3391 raft_consensus.cc:513] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:53.999210 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f76dbd7f681b418ca7bb6c6e8f1f1e58 (table default.Table_ebead212ebdd45e29399da06bdace3c8 [id=f3be079a52354875950d1c3b9ebec8bf]) successfully deleted
I20250112 20:51:53.999593 1185 tablet_replica.cc:331] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:53.999841 1185 raft_consensus.cc:2238] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:53.999904 3391 leader_election.cc:290] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.000164 1185 raft_consensus.cc:2267] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.001698 983 leader_election.cc:304] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:54.001791 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d341f6ac6848d9a8e60f689dd24a66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:54.002362 3178 consensus_queue.cc:1035] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.003121 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d341f6ac6848d9a8e60f689dd24a66" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:54.003285 1046 raft_consensus.cc:3054] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.005434 1185 ts_tablet_manager.cc:1905] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.006119 1046 raft_consensus.cc:2463] T 43d341f6ac6848d9a8e60f689dd24a66 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:54.008121 710 leader_election.cc:304] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:54.008401 3114 raft_consensus.cc:2798] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.008533 3114 raft_consensus.cc:695] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
W20250112 20:51:54.008767 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9db38abce01448dabb432c5b84b46dff consensus state has no leader
I20250112 20:51:54.008747 3114 consensus_queue.cc:237] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:54.009819 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ade3a33dfabb42499213ada871ddb7d8 consensus state has no leader
I20250112 20:51:54.010021 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 235faeb3c35d46d2882ffcd8cb63ddb8 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.010735 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_53972fb4ced04252bb122116083a5c9b" } new_table_name: "default.Table_7eea49abf85a462eaf63237200af9889" modify_external_catalogs: true
W20250112 20:51:54.014621 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1aa4249c43bb41fe8dec95e0c910a2c5 consensus state has no leader
I20250112 20:51:54.015515 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_d67e1b63833b4b2194ce174caebc8832"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.015965 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67e1b63833b4b2194ce174caebc8832 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.017187 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_6763d75a41e34bc7aee13341771b4c7e" } new_table_name: "default.Table_667d1d5b9088499bb0d7555d09988c3c" modify_external_catalogs: true
W20250112 20:51:54.017802 710 leader_election.cc:343] T 43d341f6ac6848d9a8e60f689dd24a66 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.018574 2447 ts_tablet_manager.cc:1918] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.018708 2447 log.cc:1198] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/5f264d84ae4846d1afaaabc41d5789f4
I20250112 20:51:54.018910 3130 raft_consensus.cc:3054] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.019138 2447 ts_tablet_manager.cc:1939] T 5f264d84ae4846d1afaaabc41d5789f4 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.019322 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_36a2197ce83e46658fc81ee651854f25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.019541 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1065) took 3716 ms. Trace:
I20250112 20:51:54.019620 2447 rpcz_store.cc:276] 0112 20:51:50.303079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.303140 (+ 61us) service_pool.cc:225] Handling call
0112 20:51:53.913524 (+3610384us) ts_tablet_manager.cc:1145] Deleting tablet 5f264d84ae4846d1afaaabc41d5789f4
0112 20:51:53.913644 (+ 120us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:53.973797 (+ 60153us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.018459 (+ 44662us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.019524 (+ 1065us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3592509}
W20250112 20:51:54.019753 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36a2197ce83e46658fc81ee651854f25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.020028 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1066) took 3689 ms. Trace:
I20250112 20:51:54.020090 2447 rpcz_store.cc:276] 0112 20:51:50.330304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.330367 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:54.019874 (+3689507us) ts_tablet_manager.cc:1145] Deleting tablet f7b722b0eb5944a4b344a3833905a2b0
0112 20:51:54.019884 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.020013 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3689309}
I20250112 20:51:54.020340 2447 tablet_replica.cc:331] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.020538 2447 raft_consensus.cc:2238] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.020843 2447 pending_rounds.cc:62] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:54.020942 2447 pending_rounds.cc:69] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7113585092518305792 op_type: NO_OP noop_request { }
I20250112 20:51:54.020926 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f9a8669941f404bada7ff43ad88e1a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.021061 2447 raft_consensus.cc:2883] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:54.021144 2447 raft_consensus.cc:2267] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:54.021360 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f9a8669941f404bada7ff43ad88e1a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.021531 3130 raft_consensus.cc:2743] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.021693 2447 ts_tablet_manager.cc:1905] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.022471 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 5f264d84ae4846d1afaaabc41d5789f4 (table default.Table_714de3355adc4778a417b60c786e8e4a [id=ebe9a21860414f09a21bfa8d038e2b39]) successfully deleted
W20250112 20:51:54.022823 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet f7b722b0eb5944a4b344a3833905a2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b722b0eb5944a4b344a3833905a2b0 already in progress: creating tablet
I20250112 20:51:54.025846 1163 ts_tablet_manager.cc:1428] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20250112 20:51:54.026546 1163 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.028573 1163 tablet_bootstrap.cc:654] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.032020 3476 consensus_queue.cc:1035] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.035470 1163 tablet_bootstrap.cc:492] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.035724 1163 ts_tablet_manager.cc:1397] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20250112 20:51:54.045094 1163 raft_consensus.cc:357] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.045341 1163 raft_consensus.cc:383] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.045436 1163 raft_consensus.cc:738] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.045969 3178 raft_consensus.cc:491] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.046116 3178 raft_consensus.cc:513] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.046860 3178 leader_election.cc:290] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:54.045684 1163 consensus_queue.cc:260] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.048029 1163 ts_tablet_manager.cc:1428] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250112 20:51:54.048244 446 catalog_manager.cc:5526] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aefe1db8b19244d891b6bcd8971e9b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:54.048624 446 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 263b370f96044305a79ebdb79d5e01d1 (table default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4]). Expected version 1 got 0
I20250112 20:51:54.048871 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_cb59ccaa39f84b90b998e2cee469100b [id=3ddc179c72f74f78b0ec696d624429c4] alter complete (version 3)
I20250112 20:51:54.049919 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_aa8e6a73adec405db977cedb31479617" } modify_external_catalogs: true
I20250112 20:51:54.050065 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_aa8e6a73adec405db977cedb31479617" } modify_external_catalogs: true
I20250112 20:51:54.051795 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_84f9735e457340a490e02f9c6983d2e5" } modify_external_catalogs: true
I20250112 20:51:54.051924 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_84f9735e457340a490e02f9c6983d2e5" } modify_external_catalogs: true
I20250112 20:51:54.054571 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 008d69d199d64bdf85da17d8d3bf6264 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:51 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.054837 1186 tablet_replica.cc:331] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.054816 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.055035 1186 raft_consensus.cc:2238] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.055191 1186 raft_consensus.cc:2267] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.055534 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b4b312c558437892388344767f0ed4 with delete_type TABLET_DATA_DELETED (replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.055717 1186 ts_tablet_manager.cc:1905] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.055750 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.056006 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c with delete_type TABLET_DATA_DELETED (replaced by tablet dfc63eff44414906a027a8208b5cc5a9) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.056339 3373 raft_consensus.cc:491] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.056454 3373 raft_consensus.cc:513] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.057045 3373 leader_election.cc:290] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.057940 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f425779195594599a3ab8c4dc5bee7f9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.058158 901 raft_consensus.cc:2463] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:54.058868 983 leader_election.cc:304] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:54.059180 3373 raft_consensus.cc:2798] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.059262 3373 raft_consensus.cc:491] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.059319 3373 raft_consensus.cc:3054] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.059558 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 691e2c3f72ab4f439678d166b9b31661 with delete_type TABLET_DATA_DELETED (replaced by tablet 69d57977510c4422ad625a7045ede7e4) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.028767 1185 ts_tablet_manager.cc:1918] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.060973 1185 log.cc:1198] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9880e7d303b54d959bca4e8464a6b30c
I20250112 20:51:54.061422 1185 ts_tablet_manager.cc:1939] T 9880e7d303b54d959bca4e8464a6b30c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.061861 3373 raft_consensus.cc:513] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.062243 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1094) took 4163 ms. Trace:
I20250112 20:51:54.062345 1185 rpcz_store.cc:276] 0112 20:51:49.898343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.915991 (+ 17648us) service_pool.cc:225] Handling call
0112 20:51:53.999536 (+4083545us) ts_tablet_manager.cc:1145] Deleting tablet 9880e7d303b54d959bca4e8464a6b30c
0112 20:51:53.999550 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.025445 (+ 25895us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.028685 (+ 3240us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.062225 (+ 33540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4083329}
I20250112 20:51:54.062445 3373 leader_election.cc:290] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.062753 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9880e7d303b54d959bca4e8464a6b30c (table default.Table_0cf0e2c39db54427949f7adfbec95e0d [id=537bf8b60c0249efb50df78260ed60c3]) successfully deleted
I20250112 20:51:54.063115 1185 tablet_replica.cc:331] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.063340 1185 raft_consensus.cc:2238] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.063534 1185 raft_consensus.cc:2267] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.063519 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f425779195594599a3ab8c4dc5bee7f9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:54.063841 901 raft_consensus.cc:3054] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.064146 1185 ts_tablet_manager.cc:1905] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.070799 983 consensus_peers.cc:487] T ab9be1f065ec4605a0a3fe69770793d0 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab9be1f065ec4605a0a3fe69770793d0. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:54.071226 3470 raft_consensus.cc:491] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 87ae45957c64469d96dc92d606b5b4b3)
I20250112 20:51:54.071385 3470 raft_consensus.cc:513] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.072039 3470 leader_election.cc:290] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:54.072156 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 630427adafd247578a31c900fa242cdd consensus state has no leader
I20250112 20:51:54.072150 3470 raft_consensus.cc:491] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.072307 2447 ts_tablet_manager.cc:1918] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250112 20:51:54.072253 3470 raft_consensus.cc:513] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.072403 2447 log.cc:1198] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/1ca8fc4a45fe448b919ac001a7980deb
I20250112 20:51:54.072726 3470 leader_election.cc:290] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.073400 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f425779195594599a3ab8c4dc5bee7f9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.074200 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f425779195594599a3ab8c4dc5bee7f9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:54.074474 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994d41a7ecd418a98a3c203add0d4d4" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.075913 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b4b312c558437892388344767f0ed4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:54.076560 983 leader_election.cc:343] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.043109 1164 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.077021 1164 ts_tablet_manager.cc:1397] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.145s user 0.004s sys 0.000s
W20250112 20:51:54.077157 983 leader_election.cc:343] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.077734 1164 raft_consensus.cc:357] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.077916 1164 raft_consensus.cc:383] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.077973 1164 raft_consensus.cc:738] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:54.078081 710 leader_election.cc:343] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Not found: Tablet not found: 53551931b23d43da89e9193cdf034a63
I20250112 20:51:54.078157 1164 consensus_queue.cc:260] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:54.078511 710 leader_election.cc:343] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.078649 1164 ts_tablet_manager.cc:1428] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:54.078900 1164 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.080245 1164 tablet_bootstrap.cc:654] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.048342 1163 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.084070 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet d72ae28879dc45429de2ab0d66276255 (table default.Table_850a0d99c3da4d0cbcf6d6ebf55f49bf [id=9a4f3120210440ea957d92ed21582da8]). Expected version 2 got 0
I20250112 20:51:54.084259 450 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 720aa779aada46d5a888d5011804a6a8 (table default.Table_55404e958b16400f9a4f336dc5f3cd32 [id=ff62e1f5b1aa45408d49878b0c050510]). Expected version 1 got 0
I20250112 20:51:54.085430 1163 tablet_bootstrap.cc:654] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.085760 450 master_service.cc:946] ReplaceTablet: received request to replace tablet dfc63eff44414906a027a8208b5cc5a9 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.086097 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994d41a7ecd418a98a3c203add0d4d4" candidate_uuid: "87ae45957c64469d96dc92d606b5b4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:54.086326 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dface6a2477f47e893a51ed847320520" } modify_external_catalogs: true
I20250112 20:51:54.086488 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dface6a2477f47e893a51ed847320520" } modify_external_catalogs: true
I20250112 20:51:54.086836 1186 ts_tablet_manager.cc:1918] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.086951 1186 log.cc:1198] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/008d69d199d64bdf85da17d8d3bf6264
I20250112 20:51:54.086961 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b4b312c558437892388344767f0ed4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.087164 913 raft_consensus.cc:2463] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 0.
I20250112 20:51:54.087258 1186 ts_tablet_manager.cc:1939] T 008d69d199d64bdf85da17d8d3bf6264 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.087569 710 leader_election.cc:304] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:54.087661 1186 tablet_replica.cc:331] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.088160 3470 raft_consensus.cc:2798] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.088189 901 raft_consensus.cc:2463] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:54.088255 3470 raft_consensus.cc:491] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.088325 3470 raft_consensus.cc:3054] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.088291 1186 raft_consensus.cc:2238] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.088536 1186 raft_consensus.cc:2267] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:54.088625 847 leader_election.cc:343] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.088923 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 008d69d199d64bdf85da17d8d3bf6264 (table default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]) successfully deleted
W20250112 20:51:54.089447 847 leader_election.cc:343] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.089538 847 leader_election.cc:304] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:54.089664 1186 ts_tablet_manager.cc:1905] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.089830 3305 raft_consensus.cc:2743] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.091641 3470 raft_consensus.cc:513] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.092391 3470 leader_election.cc:290] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.092612 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994d41a7ecd418a98a3c203add0d4d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:37 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.093194 983 leader_election.cc:304] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:54.094075 710 leader_election.cc:343] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 53551931b23d43da89e9193cdf034a63
I20250112 20:51:54.094105 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b4b312c558437892388344767f0ed4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:54.094281 912 raft_consensus.cc:3054] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.094333 710 leader_election.cc:304] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.094664 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b4b312c558437892388344767f0ed4" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:54.095971 2447 ts_tablet_manager.cc:1939] T 1ca8fc4a45fe448b919ac001a7980deb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:54.096822 824 tablet.cc:2367] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:54.098052 3373 raft_consensus.cc:2798] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.098255 3373 raft_consensus.cc:695] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:54.098532 3373 consensus_queue.cc:237] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.099221 912 raft_consensus.cc:2463] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:54.102823 879 tablet_service.cc:1467] Processing CreateTablet for tablet 032fc8ec35694a43885249321da1f215 (DEFAULT_TABLE table=default.Table_2627bae62eb34d008e734cba137ae9b5 [id=d9982f4d50de42ca8a93fb6936638789]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.103206 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032fc8ec35694a43885249321da1f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.104921 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1063) took 3802 ms. Trace:
I20250112 20:51:54.105079 2447 rpcz_store.cc:276] 0112 20:51:50.302354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.303519 (+ 1165us) service_pool.cc:225] Handling call
0112 20:51:54.020297 (+3716778us) ts_tablet_manager.cc:1145] Deleting tablet 1ca8fc4a45fe448b919ac001a7980deb
0112 20:51:54.020305 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.069105 (+ 48800us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.072230 (+ 3125us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.104893 (+ 32663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3686078}
I20250112 20:51:54.105398 2447 tablet_replica.cc:331] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.105597 2447 raft_consensus.cc:2238] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.105762 2447 raft_consensus.cc:2267] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.106367 2447 ts_tablet_manager.cc:1905] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.107688 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d4bd27cc30c422d85470c5a2df5d3ab consensus state has no leader
I20250112 20:51:54.108259 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 235faeb3c35d46d2882ffcd8cb63ddb8
I20250112 20:51:54.108634 455 catalog_manager.cc:6596] ReplaceTablet: tablet 235faeb3c35d46d2882ffcd8cb63ddb8 deleted and replaced by tablet a2f3475417ea4698aa6b52b0a894092d
I20250112 20:51:54.108144 1186 ts_tablet_manager.cc:1918] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.109406 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 1ca8fc4a45fe448b919ac001a7980deb (table default.Table_a1454445414a4f2882deedb81bad67bf [id=a40bd70a54ec4854a36809ac58df29b0]) successfully deleted
I20250112 20:51:54.109462 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f3475417ea4698aa6b52b0a894092d) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.110231 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f3475417ea4698aa6b52b0a894092d) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.109262 1186 log.cc:1198] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/f425779195594599a3ab8c4dc5bee7f9
I20250112 20:51:54.112380 3470 raft_consensus.cc:2743] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:54.112565 1186 ts_tablet_manager.cc:1939] T f425779195594599a3ab8c4dc5bee7f9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.112915 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 235faeb3c35d46d2882ffcd8cb63ddb8 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f3475417ea4698aa6b52b0a894092d) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.113633 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_efaca947083e41de97dc314d1610ffad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.114059 710 leader_election.cc:343] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:54.114190 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efaca947083e41de97dc314d1610ffad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.114547 710 leader_election.cc:304] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3
I20250112 20:51:54.115221 3470 raft_consensus.cc:2798] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.115347 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_febe7e80fab0422da18aa05a5609d2f0" } modify_external_catalogs: true
I20250112 20:51:54.115471 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_febe7e80fab0422da18aa05a5609d2f0" } modify_external_catalogs: true
I20250112 20:51:54.115574 3470 raft_consensus.cc:695] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Becoming Leader. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Running, Role: LEADER
I20250112 20:51:54.115392 1186 tablet_replica.cc:331] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.115811 3470 consensus_queue.cc:237] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.116937 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet f425779195594599a3ab8c4dc5bee7f9 (table default.Table_1e49d5313296492aa6576d85862799b6 [id=321d7f3949b14610affd81573168873a]) successfully deleted
I20250112 20:51:54.116838 1186 raft_consensus.cc:2238] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.117633 3476 raft_consensus.cc:491] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.117794 3476 raft_consensus.cc:513] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.117904 1186 raft_consensus.cc:2267] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.118446 3476 leader_election.cc:290] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:54.119765 983 leader_election.cc:343] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:54.120117 1185 ts_tablet_manager.cc:1918] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.120532 1185 log.cc:1198] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/f875f075e98e45499291b519caf03184
I20250112 20:51:54.125208 1186 ts_tablet_manager.cc:1905] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.121371 1164 tablet_bootstrap.cc:492] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.125851 1164 ts_tablet_manager.cc:1397] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250112 20:51:54.126629 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 032fc8ec35694a43885249321da1f215 (DEFAULT_TABLE table=default.Table_2627bae62eb34d008e734cba137ae9b5 [id=d9982f4d50de42ca8a93fb6936638789]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.126612 1164 raft_consensus.cc:357] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.126785 1164 raft_consensus.cc:383] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.126832 1164 raft_consensus.cc:738] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.126991 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032fc8ec35694a43885249321da1f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.127005 1164 consensus_queue.cc:260] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.127401 1164 ts_tablet_manager.cc:1428] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:54.127625 1164 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.129019 1164 tablet_bootstrap.cc:654] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.129920 1185 ts_tablet_manager.cc:1939] T f875f075e98e45499291b519caf03184 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.130264 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1101) took 4229 ms. Trace:
I20250112 20:51:54.130380 1185 rpcz_store.cc:276] 0112 20:51:49.900762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.916323 (+ 15561us) service_pool.cc:225] Handling call
0112 20:51:54.063064 (+4146741us) ts_tablet_manager.cc:1145] Deleting tablet f875f075e98e45499291b519caf03184
0112 20:51:54.063073 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.067152 (+ 4079us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.120027 (+ 52875us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.130250 (+ 10223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4146539}
I20250112 20:51:54.130900 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet f875f075e98e45499291b519caf03184 (table default.Table_0a39583e302f435bb0ae3f12b56e8e51 [id=a41262782d3c41bea2fc1decd8ba5a95]) successfully deleted
I20250112 20:51:54.130978 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1087) took 4262 ms. Trace:
I20250112 20:51:54.131304 1185 rpcz_store.cc:276] 0112 20:51:49.868521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.941120 (+ 72599us) service_pool.cc:225] Handling call
0112 20:51:54.130864 (+4189744us) ts_tablet_manager.cc:1145] Deleting tablet ced81345e31a4634bb21e3702ad69c93
0112 20:51:54.130873 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.130964 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4189436}
W20250112 20:51:54.131773 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet ced81345e31a4634bb21e3702ad69c93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ced81345e31a4634bb21e3702ad69c93 already in progress: creating tablet
I20250112 20:51:54.132426 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1102) took 4188 ms. Trace:
I20250112 20:51:54.132557 1185 rpcz_store.cc:276] 0112 20:51:49.943605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.944829 (+ 1224us) service_pool.cc:225] Handling call
0112 20:51:54.132332 (+4187503us) ts_tablet_manager.cc:1145] Deleting tablet a5819f1c6329433183492855c3b08476
0112 20:51:54.132341 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.132410 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4187227}
W20250112 20:51:54.133028 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a5819f1c6329433183492855c3b08476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5819f1c6329433183492855c3b08476 already in progress: creating tablet
I20250112 20:51:54.133473 1185 tablet_replica.cc:331] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.138522 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_f67e590a5132441b853dd94c1efbfbdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.139117 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f67e590a5132441b853dd94c1efbfbdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.139725 2447 ts_tablet_manager.cc:1918] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:54.139815 2447 log.cc:1198] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/db9178cf782d4bd28eba90cc03796885
I20250112 20:51:54.140151 2447 ts_tablet_manager.cc:1939] T db9178cf782d4bd28eba90cc03796885 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.140473 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1061) took 3847 ms. Trace:
I20250112 20:51:54.140556 2447 rpcz_store.cc:276] 0112 20:51:50.292587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.303661 (+ 11074us) service_pool.cc:225] Handling call
0112 20:51:54.105356 (+3801695us) ts_tablet_manager.cc:1145] Deleting tablet db9178cf782d4bd28eba90cc03796885
0112 20:51:54.105365 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.109736 (+ 4371us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.139633 (+ 29897us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.140454 (+ 821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3767194}
I20250112 20:51:54.140616 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_815f2be38a9241d6a8a478e8babdf813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.141165 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_815f2be38a9241d6a8a478e8babdf813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.146028 749 tablet_service.cc:1467] Processing CreateTablet for tablet 032fc8ec35694a43885249321da1f215 (DEFAULT_TABLE table=default.Table_2627bae62eb34d008e734cba137ae9b5 [id=d9982f4d50de42ca8a93fb6936638789]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.146406 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032fc8ec35694a43885249321da1f215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.146574 879 tablet_service.cc:1467] Processing CreateTablet for tablet ff0d0d92ecb84784816ac9fd698b32ec (DEFAULT_TABLE table=default.Table_971f4e13c7144ff0b72e5b6e65629056 [id=cfcfc041b68d45c6ade6f16d5f74654c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.146914 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0d0d92ecb84784816ac9fd698b32ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.148945 1164 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.149076 1164 ts_tablet_manager.cc:1397] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20250112 20:51:54.149029 748 tablet_service.cc:1467] Processing CreateTablet for tablet ff0d0d92ecb84784816ac9fd698b32ec (DEFAULT_TABLE table=default.Table_971f4e13c7144ff0b72e5b6e65629056 [id=cfcfc041b68d45c6ade6f16d5f74654c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.149391 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0d0d92ecb84784816ac9fd698b32ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.149724 1164 raft_consensus.cc:357] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.149911 1164 raft_consensus.cc:383] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.149956 1164 raft_consensus.cc:738] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.150151 1164 consensus_queue.cc:260] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.150494 1164 ts_tablet_manager.cc:1428] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:54.150712 1164 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:54.151300 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 630427adafd247578a31c900fa242cdd (table default.Table_7eea49abf85a462eaf63237200af9889 [id=f7e31439aed748ebbf1556c8ef59addf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 630427adafd247578a31c900fa242cdd consensus state has no leader
I20250112 20:51:54.152010 1164 tablet_bootstrap.cc:654] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.153023 756 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb (DEFAULT_TABLE table=default.Table_2e8a11571feb4678a59c17ef38959d7e [id=3d0cd45b9d0b489097bf3a4e18fe942d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.153343 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.154098 2447 tablet_replica.cc:331] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.154302 2447 raft_consensus.cc:2238] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.154438 2447 raft_consensus.cc:2267] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.154913 2447 ts_tablet_manager.cc:1905] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.157255 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]) successfully deleted
I20250112 20:51:54.157378 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ff0d0d92ecb84784816ac9fd698b32ec (DEFAULT_TABLE table=default.Table_971f4e13c7144ff0b72e5b6e65629056 [id=cfcfc041b68d45c6ade6f16d5f74654c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.158691 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 02264625e0d9441882ecefcc8cc36227
I20250112 20:51:54.159390 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb (DEFAULT_TABLE table=default.Table_2e8a11571feb4678a59c17ef38959d7e [id=3d0cd45b9d0b489097bf3a4e18fe942d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.163189 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb (DEFAULT_TABLE table=default.Table_2e8a11571feb4678a59c17ef38959d7e [id=3d0cd45b9d0b489097bf3a4e18fe942d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.163414 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 02264625e0d9441882ecefcc8cc36227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.163511 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.164458 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 64d53fc6853d4562b197b99eefa4fce0 (DEFAULT_TABLE table=default.Table_4d8793ff73b749f1a8e278bd1edcba89 [id=50b91c010c894dc2a75c906a46407f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.164867 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff0d0d92ecb84784816ac9fd698b32ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.165004 885 tablet_service.cc:1467] Processing CreateTablet for tablet 64d53fc6853d4562b197b99eefa4fce0 (DEFAULT_TABLE table=default.Table_4d8793ff73b749f1a8e278bd1edcba89 [id=50b91c010c894dc2a75c906a46407f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.165077 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c796c2a9e4b079a7f2c58e3b61aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.165225 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 02264625e0d9441882ecefcc8cc36227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.165277 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53fc6853d4562b197b99eefa4fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.165302 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53fc6853d4562b197b99eefa4fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.168874 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 02264625e0d9441882ecefcc8cc36227 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:53 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.171655 749 tablet_service.cc:1467] Processing CreateTablet for tablet 64d53fc6853d4562b197b99eefa4fce0 (DEFAULT_TABLE table=default.Table_4d8793ff73b749f1a8e278bd1edcba89 [id=50b91c010c894dc2a75c906a46407f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.172039 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53fc6853d4562b197b99eefa4fce0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:54.172878 710 consensus_peers.cc:487] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d167065029a46b791d413db8a2c4bcc. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:54.174106 891 tablet_service.cc:1467] Processing CreateTablet for tablet 54a3214284cd4a2d9141e43d32980161 (DEFAULT_TABLE table=default.Table_2546fb715e33431b915fe26d967b24c7 [id=e0157b7b8b4f4d8aae1a429d8032d267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.174468 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a3214284cd4a2d9141e43d32980161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.181633 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.184268 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:54.185499 449 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 263b370f96044305a79ebdb79d5e01d1 (table default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4]). Expected version 1 got 0
I20250112 20:51:54.186991 1163 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.187144 1163 ts_tablet_manager.cc:1397] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.139s user 0.001s sys 0.002s
I20250112 20:51:54.187964 1163 raft_consensus.cc:357] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.188190 1163 raft_consensus.cc:383] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.188269 1163 raft_consensus.cc:738] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.188546 1163 consensus_queue.cc:260] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.191747 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet b30a34c1abcb4f72a7f4f8a8f5be3720 with delete_type TABLET_DATA_DELETED (replaced by tablet 782a195e68124cd391d0394bcb5ec8b2) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.191944 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.192103 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.192755 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_089393cb2b27493a9603c9a1ec1c8959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.193312 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_089393cb2b27493a9603c9a1ec1c8959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.193581 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca8fc4a45fe448b919ac001a7980deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.195130 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cbe4933f2b49ebbdb6eb7f0f8db088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:36 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.196462 1164 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.196645 1164 ts_tablet_manager.cc:1397] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250112 20:51:54.197314 1164 raft_consensus.cc:357] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.197546 1164 raft_consensus.cc:383] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.197630 1164 raft_consensus.cc:738] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.197851 1164 consensus_queue.cc:260] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.198316 1164 ts_tablet_manager.cc:1428] T 3994d41a7ecd418a98a3c203add0d4d4 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:54.198625 1164 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.199999 1164 tablet_bootstrap.cc:654] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.201143 446 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 720aa779aada46d5a888d5011804a6a8 (table default.Table_55404e958b16400f9a4f336dc5f3cd32 [id=ff62e1f5b1aa45408d49878b0c050510]). Expected version 1 got 0
I20250112 20:51:54.202661 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_0832025535094086bbf458f978aef1ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.203269 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0832025535094086bbf458f978aef1ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.204450 1039 raft_consensus.cc:3054] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.205879 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 54a3214284cd4a2d9141e43d32980161 (DEFAULT_TABLE table=default.Table_2546fb715e33431b915fe26d967b24c7 [id=e0157b7b8b4f4d8aae1a429d8032d267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.206494 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a3214284cd4a2d9141e43d32980161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.208297 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:54.209805 748 tablet_service.cc:1467] Processing CreateTablet for tablet 54a3214284cd4a2d9141e43d32980161 (DEFAULT_TABLE table=default.Table_2546fb715e33431b915fe26d967b24c7 [id=e0157b7b8b4f4d8aae1a429d8032d267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.210193 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a3214284cd4a2d9141e43d32980161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.210552 888 tablet_service.cc:1467] Processing CreateTablet for tablet 998d7a5e2a964d19ab0be296873a9632 (DEFAULT_TABLE table=default.Table_e66533184b3d4477be0b0e3ac0da4b60 [id=eafe2d8b3cfa4a51a01f114f5e30900d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.210911 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998d7a5e2a964d19ab0be296873a9632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.210961 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 998d7a5e2a964d19ab0be296873a9632 (DEFAULT_TABLE table=default.Table_e66533184b3d4477be0b0e3ac0da4b60 [id=eafe2d8b3cfa4a51a01f114f5e30900d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.211277 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998d7a5e2a964d19ab0be296873a9632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.213142 756 tablet_service.cc:1467] Processing CreateTablet for tablet 998d7a5e2a964d19ab0be296873a9632 (DEFAULT_TABLE table=default.Table_e66533184b3d4477be0b0e3ac0da4b60 [id=eafe2d8b3cfa4a51a01f114f5e30900d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.213482 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998d7a5e2a964d19ab0be296873a9632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.214864 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 42c88852d29545c7a937ca389929d2b1
W20250112 20:51:54.215314 1097 tablet.cc:2367] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:54.216923 2447 ts_tablet_manager.cc:1918] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.216938 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c88852d29545c7a937ca389929d2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.217017 2447 log.cc:1198] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:54.217545 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c88852d29545c7a937ca389929d2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.218180 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c88852d29545c7a937ca389929d2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.218323 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_fe33d2b2bc504aa69fc27c1df3822df1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.218596 2447 ts_tablet_manager.cc:1939] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
W20250112 20:51:54.218794 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe33d2b2bc504aa69fc27c1df3822df1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.218906 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1067) took 3877 ms. Trace:
I20250112 20:51:54.218981 2447 rpcz_store.cc:276] 0112 20:51:50.341853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.342439 (+ 586us) service_pool.cc:225] Handling call
0112 20:51:54.154053 (+3811614us) ts_tablet_manager.cc:1145] Deleting tablet b85b9e82f9e847c4acfeecb82201e694
0112 20:51:54.154063 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.187396 (+ 33333us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.203607 (+ 16211us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.218891 (+ 15284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3763677}
I20250112 20:51:54.219362 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1073) took 3388 ms. Trace:
I20250112 20:51:54.219434 2447 rpcz_store.cc:276] 0112 20:51:50.831278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.831421 (+ 143us) service_pool.cc:225] Handling call
0112 20:51:54.219235 (+3387814us) ts_tablet_manager.cc:1145] Deleting tablet 886690f24fa540a6b496c926bc7a2b07
0112 20:51:54.219243 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.219344 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3387570}
I20250112 20:51:54.219741 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1074) took 3387 ms. Trace:
I20250112 20:51:54.219813 2447 rpcz_store.cc:276] 0112 20:51:50.832808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.832948 (+ 140us) service_pool.cc:225] Handling call
0112 20:51:54.219638 (+3386690us) ts_tablet_manager.cc:1145] Deleting tablet 056ed0fd01824bcea53d58251328898c
0112 20:51:54.219649 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.219729 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3386437}
I20250112 20:51:54.220093 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1075) took 3294 ms. Trace:
I20250112 20:51:54.220161 2447 rpcz_store.cc:276] 0112 20:51:50.925919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.925974 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:54.220002 (+3294028us) ts_tablet_manager.cc:1145] Deleting tablet 096ee7d3a66846f9b619d8f2fc046616
0112 20:51:54.220009 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.220079 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3293842}
I20250112 20:51:54.220460 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1076) took 3216 ms. Trace:
I20250112 20:51:54.220535 2447 rpcz_store.cc:276] 0112 20:51:51.004048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.004101 (+ 53us) service_pool.cc:225] Handling call
0112 20:51:54.220374 (+3216273us) ts_tablet_manager.cc:1145] Deleting tablet 8074b00eda8d434faaff4b98ae923ebe
0112 20:51:54.220381 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.220444 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216072}
I20250112 20:51:54.221309 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet b85b9e82f9e847c4acfeecb82201e694 (table default.Table_7927297ff7264be5a6391d2a58f38280 [id=8530d1e179394e18ad4464ae2187dcf9]) successfully deleted
W20250112 20:51:54.221547 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 886690f24fa540a6b496c926bc7a2b07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886690f24fa540a6b496c926bc7a2b07 already in progress: creating tablet
W20250112 20:51:54.221740 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 056ed0fd01824bcea53d58251328898c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056ed0fd01824bcea53d58251328898c already in progress: creating tablet
W20250112 20:51:54.221905 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 096ee7d3a66846f9b619d8f2fc046616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 096ee7d3a66846f9b619d8f2fc046616 already in progress: creating tablet
W20250112 20:51:54.222246 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 8074b00eda8d434faaff4b98ae923ebe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8074b00eda8d434faaff4b98ae923ebe already in progress: creating tablet
I20250112 20:51:54.222816 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4beae7a12751469aa5df64102bcfe38f (DEFAULT_TABLE table=default.Table_e1ffb6b5445f4fedb10499f1f7a32e3a [id=5fe39e21152d4b938c7918658f45e106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.223243 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beae7a12751469aa5df64102bcfe38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.223120 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_10b196de256a48979affcdea05019df2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.223577 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10b196de256a48979affcdea05019df2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.225054 3476 raft_consensus.cc:491] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.225222 3476 raft_consensus.cc:513] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.225518 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4beae7a12751469aa5df64102bcfe38f (DEFAULT_TABLE table=default.Table_e1ffb6b5445f4fedb10499f1f7a32e3a [id=5fe39e21152d4b938c7918658f45e106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.225826 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beae7a12751469aa5df64102bcfe38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.225999 3476 leader_election.cc:290] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.228816 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 023ab1730f1a4abbaca73018d518b1a3
W20250112 20:51:54.229182 983 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 65784493c6ef417993b1f6371cab47d6
I20250112 20:51:54.232715 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.232853 1178 tablet_bootstrap.cc:492] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:54.232986 1178 ts_tablet_manager.cc:1397] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.367s user 0.030s sys 0.048s
I20250112 20:51:54.233724 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_8d9f63102f7546f9b40db699c7d49be9" } new_table_name: "default.Table_80a02e425eee4c4da9d5154b29001249" modify_external_catalogs: true
I20250112 20:51:54.233763 1178 raft_consensus.cc:357] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.233913 1178 raft_consensus.cc:383] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.233970 1178 raft_consensus.cc:738] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.234198 1178 consensus_queue.cc:260] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.234876 1178 ts_tablet_manager.cc:1428] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:54.234913 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.241024 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1077) took 2882 ms. Trace:
I20250112 20:51:54.242738 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3b85e362c27540999daa8006a4ed60e4 (DEFAULT_TABLE table=default.Table_854e574d01624d3aa32c5cfef84ae0e8 [id=3a6d1b404a9b46889bca94391bc1ea0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.243014 737 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f (DEFAULT_TABLE table=default.Table_75d83b84b05842c28d2a5c7aa69860c5 [id=31f53e681c0c4d46891e8499be467738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.243247 2447 rpcz_store.cc:276] 0112 20:51:51.359029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.369110 (+ 10081us) service_pool.cc:225] Handling call
0112 20:51:54.220720 (+2851610us) ts_tablet_manager.cc:1145] Deleting tablet d571de1e2b104ba480cdaf2523b54599
0112 20:51:54.220729 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.241004 (+ 20275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2851355}
I20250112 20:51:54.243378 745 tablet_service.cc:1467] Processing CreateTablet for tablet 00e8f052104e4f63855281a56ed985c7 (DEFAULT_TABLE table=default.Table_d9f68cf1f11247f49b9834bfd4f94306 [id=f35067b083fc426ab187f31eacca8f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.243678 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1085) took 2685 ms. Trace:
I20250112 20:51:54.243770 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f (DEFAULT_TABLE table=default.Table_75d83b84b05842c28d2a5c7aa69860c5 [id=31f53e681c0c4d46891e8499be467738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.243829 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce7215412cf458f80cfacc7932a69d5 (DEFAULT_TABLE table=default.Table_d3bb4d64af5a47b6b91b7fbb8896c065 [id=8ba8066491eb420096d36a256b56c581]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.244096 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.244048 739 tablet_service.cc:1467] Processing CreateTablet for tablet 21ffd16ee50d42baabc85cb87b5498cf (DEFAULT_TABLE table=default.Table_b216fe36f3134924b3e2adf44426475a [id=7d434588b4de4b90964fa5aacd854624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.244824 738 tablet_service.cc:1467] Processing CreateTablet for tablet 6b67d66724e247039cef64b54c4aef07 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.245204 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e362c27540999daa8006a4ed60e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.245357 885 tablet_service.cc:1467] Processing CreateTablet for tablet 3b85e362c27540999daa8006a4ed60e4 (DEFAULT_TABLE table=default.Table_854e574d01624d3aa32c5cfef84ae0e8 [id=3a6d1b404a9b46889bca94391bc1ea0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.245561 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.243744 2447 rpcz_store.cc:276] 0112 20:51:51.557740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.565442 (+ 7702us) service_pool.cc:225] Handling call
0112 20:51:54.243570 (+2678128us) ts_tablet_manager.cc:1145] Deleting tablet b496462873e44ceab2ec3c95cf63fc5a
0112 20:51:54.243581 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.243663 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635168}
I20250112 20:51:54.245730 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce7215412cf458f80cfacc7932a69d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.245750 2447 tablet_replica.cc:331] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.245895 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ffd16ee50d42baabc85cb87b5498cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.245919 2447 raft_consensus.cc:2238] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.246100 2447 raft_consensus.cc:2267] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.246131 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b67d66724e247039cef64b54c4aef07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.246387 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f (DEFAULT_TABLE table=default.Table_75d83b84b05842c28d2a5c7aa69860c5 [id=31f53e681c0c4d46891e8499be467738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.247573 2447 ts_tablet_manager.cc:1905] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.248909 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e8f052104e4f63855281a56ed985c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.249266 884 tablet_service.cc:1467] Processing CreateTablet for tablet 00e8f052104e4f63855281a56ed985c7 (DEFAULT_TABLE table=default.Table_d9f68cf1f11247f49b9834bfd4f94306 [id=f35067b083fc426ab187f31eacca8f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.252243 739 tablet_service.cc:1467] Processing CreateTablet for tablet 2e07364f52a24b94a20a10a1863b4812 (DEFAULT_TABLE table=default.Table_0ce0b84702c84d5cb03c6761260191db [id=9838bd249eeb44b88b69a2802843b40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.252557 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e07364f52a24b94a20a10a1863b4812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.257131 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce7215412cf458f80cfacc7932a69d5 (DEFAULT_TABLE table=default.Table_d3bb4d64af5a47b6b91b7fbb8896c065 [id=8ba8066491eb420096d36a256b56c581]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.257450 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce7215412cf458f80cfacc7932a69d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:54.258029 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet d571de1e2b104ba480cdaf2523b54599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d571de1e2b104ba480cdaf2523b54599 already in progress: creating tablet
W20250112 20:51:54.258234 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet b496462873e44ceab2ec3c95cf63fc5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b496462873e44ceab2ec3c95cf63fc5a already in progress: creating tablet
I20250112 20:51:54.261961 3476 raft_consensus.cc:491] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.262135 3476 raft_consensus.cc:513] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.262795 3476 leader_election.cc:290] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.242745 756 tablet_service.cc:1467] Processing CreateTablet for tablet 4beae7a12751469aa5df64102bcfe38f (DEFAULT_TABLE table=default.Table_e1ffb6b5445f4fedb10499f1f7a32e3a [id=5fe39e21152d4b938c7918658f45e106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.263643 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beae7a12751469aa5df64102bcfe38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.264462 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6b67d66724e247039cef64b54c4aef07 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.264802 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b67d66724e247039cef64b54c4aef07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.265820 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet d8ea5b6d791d4891a1bb008879eba941
I20250112 20:51:54.266594 2447 ts_tablet_manager.cc:1918] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:54.266680 2447 log.cc:1198] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/480d49569886408a8ce781a93bb5d10d
I20250112 20:51:54.266999 2447 ts_tablet_manager.cc:1939] T 480d49569886408a8ce781a93bb5d10d P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.267253 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1086) took 2644 ms. Trace:
I20250112 20:51:54.267318 2447 rpcz_store.cc:276] 0112 20:51:51.622567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.622642 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:54.245712 (+2623070us) ts_tablet_manager.cc:1145] Deleting tablet 480d49569886408a8ce781a93bb5d10d
0112 20:51:54.245720 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.258386 (+ 12666us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.266521 (+ 8135us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.267241 (+ 720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2622879}
I20250112 20:51:54.267625 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1087) took 2594 ms. Trace:
I20250112 20:51:54.267693 2447 rpcz_store.cc:276] 0112 20:51:51.673531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.673600 (+ 69us) service_pool.cc:225] Handling call
0112 20:51:54.267530 (+2593930us) ts_tablet_manager.cc:1145] Deleting tablet a2ff6b3dbc614785a5db4b4185fe0945
0112 20:51:54.267538 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.267613 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593755}
I20250112 20:51:54.267952 2447 tablet_replica.cc:331] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.268124 2447 raft_consensus.cc:2238] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.268311 2447 pending_rounds.cc:62] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Trying to abort 1 pending ops.
I20250112 20:51:54.268344 2447 pending_rounds.cc:69] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113585106241372160 op_type: NO_OP noop_request { }
I20250112 20:51:54.268443 2447 raft_consensus.cc:2883] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 20:51:54.268501 2447 raft_consensus.cc:2267] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.268862 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ea5b6d791d4891a1bb008879eba941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.269804 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_ff0a7317912c4e05b31cac70b0d57115" } new_table_name: "default.Table_5d11989e585745d2ab1d4053acd74ea5" modify_external_catalogs: true
I20250112 20:51:54.270884 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 21ffd16ee50d42baabc85cb87b5498cf (DEFAULT_TABLE table=default.Table_b216fe36f3134924b3e2adf44426475a [id=7d434588b4de4b90964fa5aacd854624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.271322 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ffd16ee50d42baabc85cb87b5498cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.272363 2447 ts_tablet_manager.cc:1905] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.273614 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce7215412cf458f80cfacc7932a69d5 (DEFAULT_TABLE table=default.Table_d3bb4d64af5a47b6b91b7fbb8896c065 [id=8ba8066491eb420096d36a256b56c581]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.272683 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3b85e362c27540999daa8006a4ed60e4 (DEFAULT_TABLE table=default.Table_854e574d01624d3aa32c5cfef84ae0e8 [id=3a6d1b404a9b46889bca94391bc1ea0d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:54.276091 983 leader_election.cc:343] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Not found: Tablet not found: b85b9e82f9e847c4acfeecb82201e694
I20250112 20:51:54.276273 983 leader_election.cc:304] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:54.276479 983 raft_consensus.cc:2681] T b85b9e82f9e847c4acfeecb82201e694 P 30ba399c7a6749e0acd075d93e4cc4b3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:54.276535 3470 raft_consensus.cc:491] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.276095 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e362c27540999daa8006a4ed60e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.277801 3470 raft_consensus.cc:513] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.278820 3470 leader_election.cc:290] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.279670 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.279824 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6b67d66724e247039cef64b54c4aef07 (DEFAULT_TABLE table=default.Table_8d21e147bdd0405cbc9b0580dd27b827 [id=23ffed9a23794c15b7740e965da9f70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.280057 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 00e8f052104e4f63855281a56ed985c7 (DEFAULT_TABLE table=default.Table_d9f68cf1f11247f49b9834bfd4f94306 [id=f35067b083fc426ab187f31eacca8f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.280664 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e8f052104e4f63855281a56ed985c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.280700 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_e66533184b3d4477be0b0e3ac0da4b60" } new_table_name: "default.Table_c7ee485e06c649c0867622f2f9acd1da" modify_external_catalogs: true
I20250112 20:51:54.282953 1164 tablet_bootstrap.cc:492] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.282881 882 tablet_service.cc:1467] Processing CreateTablet for tablet 21ffd16ee50d42baabc85cb87b5498cf (DEFAULT_TABLE table=default.Table_b216fe36f3134924b3e2adf44426475a [id=7d434588b4de4b90964fa5aacd854624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.283143 1164 ts_tablet_manager.cc:1397] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20250112 20:51:54.283238 875 tablet_service.cc:1467] Processing CreateTablet for tablet 2e07364f52a24b94a20a10a1863b4812 (DEFAULT_TABLE table=default.Table_0ce0b84702c84d5cb03c6761260191db [id=9838bd249eeb44b88b69a2802843b40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.283444 878 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ea5b6d791d4891a1bb008879eba941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.283969 1164 raft_consensus.cc:357] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.284194 1164 raft_consensus.cc:383] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.284284 1164 raft_consensus.cc:738] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.284513 1164 consensus_queue.cc:260] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.284808 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 2e07364f52a24b94a20a10a1863b4812 (DEFAULT_TABLE table=default.Table_0ce0b84702c84d5cb03c6761260191db [id=9838bd249eeb44b88b69a2802843b40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.285029 1164 ts_tablet_manager.cc:1428] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:54.285207 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e07364f52a24b94a20a10a1863b4812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.285266 1164 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.285791 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 3b8649f8404240df8b750f431a835516
I20250112 20:51:54.286330 455 master_service.cc:946] ReplaceTablet: received request to replace tablet c2a8f71233d04ca39e49932308e56492 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.286528 1164 tablet_bootstrap.cc:654] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.288868 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8649f8404240df8b750f431a835516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.289863 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8649f8404240df8b750f431a835516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.290310 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 480d49569886408a8ce781a93bb5d10d (table default.Table_cb2b5abd46bc4877887604a94159e418 [id=d46a20ad8b99492395771d94f5dc60fb]) successfully deleted
W20250112 20:51:54.290588 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet a2ff6b3dbc614785a5db4b4185fe0945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff6b3dbc614785a5db4b4185fe0945 already in progress: creating tablet
I20250112 20:51:54.292963 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ea5b6d791d4891a1bb008879eba941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.293198 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8649f8404240df8b750f431a835516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.294098 3392 consensus_queue.cc:1035] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.297122 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65784493c6ef417993b1f6371cab47d6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.297295 763 raft_consensus.cc:2463] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:54.297537 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691e2c3f72ab4f439678d166b9b31661" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.298183 1178 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:54.301373 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05550152671e4e28829b5b102b59f715" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.301872 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b67d66724e247039cef64b54c4aef07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.302592 3392 consensus_queue.cc:1035] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.303205 1186 ts_tablet_manager.cc:1918] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.303324 1186 log.cc:1198] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/68b4b312c558437892388344767f0ed4
I20250112 20:51:54.303596 1186 ts_tablet_manager.cc:1939] T 68b4b312c558437892388344767f0ed4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.304001 1186 tablet_replica.cc:331] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.304179 1186 raft_consensus.cc:2238] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.304306 1186 raft_consensus.cc:2267] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.305176 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b85e362c27540999daa8006a4ed60e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.307435 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 68b4b312c558437892388344767f0ed4 (table default.Table_588580b3605944d0b5434d8b59053fc1 [id=3deb85d7d90c4e28a2c414618507c4f2]) successfully deleted
I20250112 20:51:54.304764 1186 ts_tablet_manager.cc:1905] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.325199 1178 tablet_bootstrap.cc:654] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:54.329903 710 leader_election.cc:343] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.330309 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0cf0e2c39db54427949f7adfbec95e0d" } new_table_name: "default.Table_771d807821e449a9ac46e5a455d42e76" modify_external_catalogs: true
I20250112 20:51:54.333341 1163 ts_tablet_manager.cc:1428] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.146s user 0.003s sys 0.000s
I20250112 20:51:54.336601 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee94a40ca5b4ff98f3278a2cee17f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.336885 756 tablet_service.cc:1467] Processing CreateTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 (DEFAULT_TABLE table=default.Table_dface6a2477f47e893a51ed847320520 [id=194e88140085419a9fbdcbe20df9bffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.336987 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e8f052104e4f63855281a56ed985c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.337226 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023ab1730f1a4abbaca73018d518b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.337502 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9516c24a4fbb4713a4bee40b895c76fa (DEFAULT_TABLE table=default.Table_a4190ff9da0d40a58ed6dac7cefcb255 [id=3a19360bfd3046beb0c6909f5de5797f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.337818 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9516c24a4fbb4713a4bee40b895c76fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.339661 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce7215412cf458f80cfacc7932a69d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.341907 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e07364f52a24b94a20a10a1863b4812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.342897 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ffd16ee50d42baabc85cb87b5498cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.344336 748 tablet_service.cc:1467] Processing CreateTablet for tablet 30e06a86f03e4ef8a5bb72be109f5a7b (DEFAULT_TABLE table=default.Table_df552fa151414d49923bc94bb418fc9f [id=8bc93723b2cf42e4801f581db30f5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.344717 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e06a86f03e4ef8a5bb72be109f5a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.348002 1164 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.348234 1164 ts_tablet_manager.cc:1397] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.000s
I20250112 20:51:54.349050 1164 raft_consensus.cc:357] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.349395 1164 raft_consensus.cc:383] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.349447 454 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 720aa779aada46d5a888d5011804a6a8 (table default.Table_55404e958b16400f9a4f336dc5f3cd32 [id=ff62e1f5b1aa45408d49878b0c050510]). Expected version 1 got 0
I20250112 20:51:54.349598 1164 raft_consensus.cc:738] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.349975 1164 consensus_queue.cc:260] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.350086 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_d1c0e87cd71d4574bbe2a7d8dee9d465" } new_table_name: "default.Table_dd58395b21654bc384e9c6831b2c65b9" modify_external_catalogs: true
I20250112 20:51:54.350809 1164 ts_tablet_manager.cc:1428] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:54.351238 1164 tablet_bootstrap.cc:492] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.353924 2447 ts_tablet_manager.cc:1918] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.355423 2447 log.cc:1198] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/008d69d199d64bdf85da17d8d3bf6264
I20250112 20:51:54.356007 2447 ts_tablet_manager.cc:1939] T 008d69d199d64bdf85da17d8d3bf6264 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.357653 1185 raft_consensus.cc:2238] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.357810 1185 raft_consensus.cc:2267] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.358275 1185 ts_tablet_manager.cc:1905] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.361848 1164 tablet_bootstrap.cc:654] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.363178 1163 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.363976 448 master_service.cc:946] ReplaceTablet: received request to replace tablet c54516970d424f7cb28c72d8fe52b824 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.365147 1163 tablet_bootstrap.cc:654] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.365469 1185 ts_tablet_manager.cc:1918] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:54.365573 1185 log.cc:1198] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/53551931b23d43da89e9193cdf034a63
I20250112 20:51:54.365898 1185 ts_tablet_manager.cc:1939] T 53551931b23d43da89e9193cdf034a63 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.366166 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1103) took 4420 ms. Trace:
I20250112 20:51:54.366649 1185 rpcz_store.cc:276] 0112 20:51:49.945821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:49.946324 (+ 503us) service_pool.cc:225] Handling call
0112 20:51:54.133433 (+4187109us) ts_tablet_manager.cc:1145] Deleting tablet 53551931b23d43da89e9193cdf034a63
0112 20:51:54.133442 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.362488 (+229046us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.365405 (+ 2917us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.366147 (+ 742us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":223932,"tablet-delete.queue_time_us":4186880}
I20250112 20:51:54.367100 1185 tablet_replica.cc:331] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.367358 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 53551931b23d43da89e9193cdf034a63 (table default.Table_ee4fb486face46d4801238c334cdfce8 [id=b685e351867f45118b3bff317a1825c1]) successfully deleted
I20250112 20:51:54.367419 1185 raft_consensus.cc:2238] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.367695 1185 raft_consensus.cc:2267] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.368216 1185 ts_tablet_manager.cc:1905] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.369024 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 72001us
W20250112 20:51:54.369282 983 leader_election.cc:343] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 691e2c3f72ab4f439678d166b9b31661
I20250112 20:51:54.369601 983 leader_election.cc:304] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.369931 3130 raft_consensus.cc:2798] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.370012 3130 raft_consensus.cc:491] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.370074 3130 raft_consensus.cc:3054] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.371366 1048 raft_consensus.cc:3054] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.372390 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05550152671e4e28829b5b102b59f715" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:54.372545 3130 raft_consensus.cc:513] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.373554 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 30e06a86f03e4ef8a5bb72be109f5a7b (DEFAULT_TABLE table=default.Table_df552fa151414d49923bc94bb418fc9f [id=8bc93723b2cf42e4801f581db30f5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.373862 1048 raft_consensus.cc:1270] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Refusing update from remote peer aefe1db8b19244d891b6bcd8971e9b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:54.374181 3130 leader_election.cc:290] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.369948 983 leader_election.cc:304] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3; no voters: 87ae45957c64469d96dc92d606b5b4b3, aefe1db8b19244d891b6bcd8971e9b17
W20250112 20:51:54.382875 983 leader_election.cc:343] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 65784493c6ef417993b1f6371cab47d6
I20250112 20:51:54.383229 1164 tablet_bootstrap.cc:492] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.383463 1164 ts_tablet_manager.cc:1397] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250112 20:51:54.384708 1164 raft_consensus.cc:357] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.385093 1164 raft_consensus.cc:383] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.385241 1164 raft_consensus.cc:738] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.385532 1164 consensus_queue.cc:260] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.386107 1185 ts_tablet_manager.cc:1918] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.388172 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_6b95e591551648e2bc3a4e42ea03c9d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.389021 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b95e591551648e2bc3a4e42ea03c9d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.386751 1164 ts_tablet_manager.cc:1428] T 2de03e3114d145aeb81590dbae2c548e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:54.390589 1164 tablet_bootstrap.cc:492] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.392958 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65784493c6ef417993b1f6371cab47d6" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:54.393121 766 raft_consensus.cc:3054] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:54.393718 710 leader_election.cc:343] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.393841 710 leader_election.cc:304] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.394040 3392 raft_consensus.cc:2743] T 05550152671e4e28829b5b102b59f715 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.394624 3392 consensus_queue.cc:1035] T 5d167065029a46b791d413db8a2c4bcc P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.395516 766 raft_consensus.cc:2463] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:54.396417 1164 tablet_bootstrap.cc:654] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.396879 3392 consensus_queue.cc:1035] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.398835 3392 raft_consensus.cc:491] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.398988 3392 raft_consensus.cc:513] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.399274 983 leader_election.cc:304] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.399605 3373 raft_consensus.cc:2798] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.399760 3373 raft_consensus.cc:695] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:54.399957 3392 leader_election.cc:290] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.400002 3373 consensus_queue.cc:237] T 65784493c6ef417993b1f6371cab47d6 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.401862 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9516c24a4fbb4713a4bee40b895c76fa (DEFAULT_TABLE table=default.Table_a4190ff9da0d40a58ed6dac7cefcb255 [id=3a19360bfd3046beb0c6909f5de5797f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.405230 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 (DEFAULT_TABLE table=default.Table_dface6a2477f47e893a51ed847320520 [id=194e88140085419a9fbdcbe20df9bffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.408018 1010 tablet_service.cc:1467] Processing CreateTablet for tablet e9020192a4d34e8eb8590c41d545efb8 (DEFAULT_TABLE table=default.Table_237a0cfba9424c13a84c153a381d7967 [id=27eafa4d49444287ad822327ddcf2e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.409533 748 tablet_service.cc:1467] Processing CreateTablet for tablet e9020192a4d34e8eb8590c41d545efb8 (DEFAULT_TABLE table=default.Table_237a0cfba9424c13a84c153a381d7967 [id=27eafa4d49444287ad822327ddcf2e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.409911 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9020192a4d34e8eb8590c41d545efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.409933 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 23a7129e527943589de5a1a28c7e1f25 (DEFAULT_TABLE table=default.Table_93ca51498c9a41faa2de22068cdb8156 [id=cd34fc61f8da40439810e3e6532cbf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.410203 756 tablet_service.cc:1467] Processing CreateTablet for tablet 23a7129e527943589de5a1a28c7e1f25 (DEFAULT_TABLE table=default.Table_93ca51498c9a41faa2de22068cdb8156 [id=cd34fc61f8da40439810e3e6532cbf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.410508 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a7129e527943589de5a1a28c7e1f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.413026 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9393e5cc434ec19bb907a6331e9d26 (DEFAULT_TABLE table=default.Table_dc131053ce3845268e1279e76cbf1ddd [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.410454 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9393e5cc434ec19bb907a6331e9d26 (DEFAULT_TABLE table=default.Table_dc131053ce3845268e1279e76cbf1ddd [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.413507 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9393e5cc434ec19bb907a6331e9d26. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:54.416857 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35585, user_credentials={real_user=slave}} blocked reactor thread for 109823us
I20250112 20:51:54.416916 1040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f425779195594599a3ab8c4dc5bee7f9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:54.417227 3304 consensus_queue.cc:1035] T d738720dd7414764b4d1191cdc37895d P 87ae45957c64469d96dc92d606b5b4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.422381 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_29629417969c412880b384cc71f77f78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.422927 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29629417969c412880b384cc71f77f78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 20:51:54.423569 710 consensus_peers.cc:487] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Couldn't send request to peer 30ba399c7a6749e0acd075d93e4cc4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 480d49569886408a8ce781a93bb5d10d. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:54.427343 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e06a86f03e4ef8a5bb72be109f5a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.427691 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1088) took 2728 ms. Trace:
I20250112 20:51:54.427773 2447 rpcz_store.cc:276] 0112 20:51:51.699234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.699311 (+ 77us) service_pool.cc:225] Handling call
0112 20:51:54.267872 (+2568561us) ts_tablet_manager.cc:1145] Deleting tablet 008d69d199d64bdf85da17d8d3bf6264
0112 20:51:54.267899 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.330874 (+ 62975us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.353833 (+ 22959us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.427675 (+ 73842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568359}
I20250112 20:51:54.428148 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1089) took 2728 ms. Trace:
I20250112 20:51:54.428220 2447 rpcz_store.cc:276] 0112 20:51:51.699356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.699555 (+ 199us) service_pool.cc:225] Handling call
0112 20:51:54.428040 (+2728485us) ts_tablet_manager.cc:1145] Deleting tablet 75790518579d496fb0efb12b497fcc75
0112 20:51:54.428052 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.428134 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728328}
I20250112 20:51:54.428445 2447 tablet_replica.cc:331] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.428619 2447 raft_consensus.cc:2238] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.428747 2447 raft_consensus.cc:2267] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.429263 2447 ts_tablet_manager.cc:1905] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.390110 1185 log.cc:1198] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/22f6b2c4ea5f4167896c78b1c64cbb62
I20250112 20:51:54.430541 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9516c24a4fbb4713a4bee40b895c76fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.431620 1185 ts_tablet_manager.cc:1939] T 22f6b2c4ea5f4167896c78b1c64cbb62 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.433612 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a7129e527943589de5a1a28c7e1f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.433281 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1108) took 4159 ms. Trace:
I20250112 20:51:54.433665 1185 rpcz_store.cc:276] 0112 20:51:50.274244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.274320 (+ 76us) service_pool.cc:225] Handling call
0112 20:51:54.367055 (+4092735us) ts_tablet_manager.cc:1145] Deleting tablet 22f6b2c4ea5f4167896c78b1c64cbb62
0112 20:51:54.367064 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.377330 (+ 10266us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.386010 (+ 8680us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.433264 (+ 47254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076104}
I20250112 20:51:54.435416 1185 tablet_replica.cc:331] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.435576 2447 ts_tablet_manager.cc:1918] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.435657 2447 log.cc:1198] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/22f6b2c4ea5f4167896c78b1c64cbb62
I20250112 20:51:54.436004 2447 ts_tablet_manager.cc:1939] T 22f6b2c4ea5f4167896c78b1c64cbb62 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.436645 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1093) took 2711 ms. Trace:
I20250112 20:51:54.436995 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9393e5cc434ec19bb907a6331e9d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.436456 1185 raft_consensus.cc:2238] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.436707 2447 rpcz_store.cc:276] 0112 20:51:51.725365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.728699 (+ 3334us) service_pool.cc:225] Handling call
0112 20:51:54.428411 (+2699712us) ts_tablet_manager.cc:1145] Deleting tablet 22f6b2c4ea5f4167896c78b1c64cbb62
0112 20:51:54.428419 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.432269 (+ 3850us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.435509 (+ 3240us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.436628 (+ 1119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2699499}
I20250112 20:51:54.438200 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1095) took 2560 ms. Trace:
I20250112 20:51:54.438261 2447 rpcz_store.cc:276] 0112 20:51:51.877629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.877689 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:54.438098 (+2560409us) ts_tablet_manager.cc:1145] Deleting tablet 417ada5111454c54a85cc95db5974bf9
0112 20:51:54.438108 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.438187 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2560213}
I20250112 20:51:54.438653 1185 raft_consensus.cc:2267] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.439419 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1096) took 2558 ms. Trace:
I20250112 20:51:54.439496 2447 rpcz_store.cc:276] 0112 20:51:51.881325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.881383 (+ 58us) service_pool.cc:225] Handling call
0112 20:51:54.439323 (+2557940us) ts_tablet_manager.cc:1145] Deleting tablet 782a195e68124cd391d0394bcb5ec8b2
0112 20:51:54.439332 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.439402 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2557775}
I20250112 20:51:54.440654 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1097) took 2517 ms. Trace:
I20250112 20:51:54.440719 2447 rpcz_store.cc:276] 0112 20:51:51.922946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.923055 (+ 109us) service_pool.cc:225] Handling call
0112 20:51:54.440563 (+2517508us) ts_tablet_manager.cc:1145] Deleting tablet df292cbdf58b4fcd889fcb63335b37d9
0112 20:51:54.440571 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.440640 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2385226}
I20250112 20:51:54.441869 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1098) took 2511 ms. Trace:
I20250112 20:51:54.442240 1185 ts_tablet_manager.cc:1905] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.442437 2447 rpcz_store.cc:276] 0112 20:51:51.929921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.929983 (+ 62us) service_pool.cc:225] Handling call
0112 20:51:54.441780 (+2511797us) ts_tablet_manager.cc:1145] Deleting tablet 91243c9c6deb447e8246f7abe2691f55
0112 20:51:54.441788 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.441854 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386379}
I20250112 20:51:54.443220 2447 tablet_replica.cc:331] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.444149 2447 raft_consensus.cc:2238] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.444633 2447 raft_consensus.cc:2267] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.445386 2447 ts_tablet_manager.cc:1905] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.446920 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_526e73f5dfe6481985f030cc69b536a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.447643 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_526e73f5dfe6481985f030cc69b536a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.451457 1164 tablet_bootstrap.cc:492] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.451651 1164 ts_tablet_manager.cc:1397] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.003s
W20250112 20:51:54.452889 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e2f3e0b9f05455090bddaa15fe0a280 consensus state has no leader
I20250112 20:51:54.452494 1164 raft_consensus.cc:357] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.455338 1164 raft_consensus.cc:383] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.455439 3470 raft_consensus.cc:491] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.455585 3470 raft_consensus.cc:513] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.455458 1164 raft_consensus.cc:738] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.456144 1164 consensus_queue.cc:260] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.456262 3470 leader_election.cc:290] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:54.457022 961 tablet.cc:2367] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:54.460105 1164 ts_tablet_manager.cc:1428] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 20:51:54.460129 1186 ts_tablet_manager.cc:1918] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.460242 1186 log.cc:1198] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/868ce9397b884b0095dae39ea57f5fd9
I20250112 20:51:54.460664 1186 ts_tablet_manager.cc:1939] T 868ce9397b884b0095dae39ea57f5fd9 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.460980 1164 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.461309 1186 tablet_replica.cc:331] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.461488 1186 raft_consensus.cc:2238] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.461637 1186 raft_consensus.cc:2267] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.462392 1186 ts_tablet_manager.cc:1905] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.464668 1185 ts_tablet_manager.cc:1918] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.465198 1185 log.cc:1198] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/9570269368e14b1abeaffdcc741c6f3a
I20250112 20:51:54.466764 1185 ts_tablet_manager.cc:1939] T 9570269368e14b1abeaffdcc741c6f3a P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.467299 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d341f6ac6848d9a8e60f689dd24a66 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d9572179a6f45759214c6c39f22b2f0) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.468024 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9516c24a4fbb4713a4bee40b895c76fa (DEFAULT_TABLE table=default.Table_a4190ff9da0d40a58ed6dac7cefcb255 [id=3a19360bfd3046beb0c6909f5de5797f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.468099 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1109) took 4178 ms. Trace:
I20250112 20:51:54.469095 879 tablet_service.cc:1467] Processing CreateTablet for tablet 023ab1730f1a4abbaca73018d518b1a3 (DEFAULT_TABLE table=default.Table_dface6a2477f47e893a51ed847320520 [id=194e88140085419a9fbdcbe20df9bffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.468519 1185 rpcz_store.cc:276] 0112 20:51:50.289330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.289429 (+ 99us) service_pool.cc:225] Handling call
0112 20:51:54.435213 (+4145784us) ts_tablet_manager.cc:1145] Deleting tablet 9570269368e14b1abeaffdcc741c6f3a
0112 20:51:54.435223 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.452886 (+ 17663us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.464125 (+ 11239us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.467984 (+ 3859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4142983}
I20250112 20:51:54.471356 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1110) took 4180 ms. Trace:
I20250112 20:51:54.471707 1185 rpcz_store.cc:276] 0112 20:51:50.291307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.291400 (+ 93us) service_pool.cc:225] Handling call
0112 20:51:54.470839 (+4179439us) ts_tablet_manager.cc:1145] Deleting tablet 8bc545822e98460e815df4e6cad455b4
0112 20:51:54.470952 (+ 113us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.471243 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4175840}
I20250112 20:51:54.473285 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1117) took 4158 ms. Trace:
I20250112 20:51:54.473428 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3ec2de0c20494221b62ee138a5073ca3" } modify_external_catalogs: true
I20250112 20:51:54.473635 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_3ec2de0c20494221b62ee138a5073ca3" } modify_external_catalogs: true
I20250112 20:51:54.473716 1185 rpcz_store.cc:276] 0112 20:51:50.314809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.315339 (+ 530us) service_pool.cc:225] Handling call
0112 20:51:54.472752 (+4157413us) ts_tablet_manager.cc:1145] Deleting tablet 5065a94bb13c419e9d9b8eaca72f5052
0112 20:51:54.472885 (+ 133us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.473176 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4157244}
I20250112 20:51:54.474555 3537 tablet.cc:1721] T 263b370f96044305a79ebdb79d5e01d1 P aefe1db8b19244d891b6bcd8971e9b17: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:54.475390 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1118) took 4160 ms. Trace:
I20250112 20:51:54.475910 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023ab1730f1a4abbaca73018d518b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.476109 892 tablet_service.cc:1467] Processing CreateTablet for tablet 30e06a86f03e4ef8a5bb72be109f5a7b (DEFAULT_TABLE table=default.Table_df552fa151414d49923bc94bb418fc9f [id=8bc93723b2cf42e4801f581db30f5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.475736 1185 rpcz_store.cc:276] 0112 20:51:50.314908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.315533 (+ 625us) service_pool.cc:225] Handling call
0112 20:51:54.475006 (+4159473us) ts_tablet_manager.cc:1145] Deleting tablet 6181b166b5cb429da3ebba23cf267942
0112 20:51:54.475109 (+ 103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.475374 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4159348}
I20250112 20:51:54.476109 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9516c24a4fbb4713a4bee40b895c76fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.476732 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e06a86f03e4ef8a5bb72be109f5a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.477051 1185 tablet_replica.cc:331] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.477813 1185 raft_consensus.cc:2238] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.478825 1185 raft_consensus.cc:2267] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.480736 1185 ts_tablet_manager.cc:1905] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.484889 1164 tablet_bootstrap.cc:654] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.485706 875 tablet_service.cc:1467] Processing CreateTablet for tablet e9020192a4d34e8eb8590c41d545efb8 (DEFAULT_TABLE table=default.Table_237a0cfba9424c13a84c153a381d7967 [id=27eafa4d49444287ad822327ddcf2e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.489053 1018 tablet_service.cc:1467] Processing CreateTablet for tablet c61749c7999d43b68dc1a44f58125fc8 (DEFAULT_TABLE table=default.Table_c65b1f79389e468fb8793d95d7248128 [id=c1d41ebadc9a4edb9ee8afe0eb739042]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:54.492509 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1aa4249c43bb41fe8dec95e0c910a2c5 consensus state has no leader
I20250112 20:51:54.492718 749 tablet_service.cc:1467] Processing CreateTablet for tablet c61749c7999d43b68dc1a44f58125fc8 (DEFAULT_TABLE table=default.Table_c65b1f79389e468fb8793d95d7248128 [id=c1d41ebadc9a4edb9ee8afe0eb739042]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.495080 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61749c7999d43b68dc1a44f58125fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.495404 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9020192a4d34e8eb8590c41d545efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.496595 891 tablet_service.cc:1467] Processing CreateTablet for tablet 23a7129e527943589de5a1a28c7e1f25 (DEFAULT_TABLE table=default.Table_93ca51498c9a41faa2de22068cdb8156 [id=cd34fc61f8da40439810e3e6532cbf11]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:54.500810 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 998d7a5e2a964d19ab0be296873a9632 consensus state has no leader
I20250112 20:51:54.509945 1185 ts_tablet_manager.cc:1918] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:54.510576 1185 log.cc:1198] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/db9178cf782d4bd28eba90cc03796885
I20250112 20:51:54.514600 1185 ts_tablet_manager.cc:1939] T db9178cf782d4bd28eba90cc03796885 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.516391 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1119) took 4201 ms. Trace:
I20250112 20:51:54.517134 1185 rpcz_store.cc:276] 0112 20:51:50.315021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.315680 (+ 659us) service_pool.cc:225] Handling call
0112 20:51:54.476902 (+4161222us) ts_tablet_manager.cc:1145] Deleting tablet db9178cf782d4bd28eba90cc03796885
0112 20:51:54.476912 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.491579 (+ 14667us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.509378 (+ 17799us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.516370 (+ 6992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4160944}
I20250112 20:51:54.519510 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1121) took 4165 ms. Trace:
I20250112 20:51:54.520134 1185 rpcz_store.cc:276] 0112 20:51:50.353709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.356834 (+ 3125us) service_pool.cc:225] Handling call
0112 20:51:54.518972 (+4162138us) ts_tablet_manager.cc:1145] Deleting tablet f7b722b0eb5944a4b344a3833905a2b0
0112 20:51:54.518985 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.519490 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4161717}
I20250112 20:51:54.522178 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1129) took 3690 ms. Trace:
I20250112 20:51:54.522728 1185 rpcz_store.cc:276] 0112 20:51:50.831342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.831402 (+ 60us) service_pool.cc:225] Handling call
0112 20:51:54.521521 (+3690119us) ts_tablet_manager.cc:1145] Deleting tablet 886690f24fa540a6b496c926bc7a2b07
0112 20:51:54.521687 (+ 166us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.522161 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3689856}
I20250112 20:51:54.524962 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1130) took 3692 ms. Trace:
I20250112 20:51:54.525606 1185 rpcz_store.cc:276] 0112 20:51:50.832879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.832943 (+ 64us) service_pool.cc:225] Handling call
0112 20:51:54.524215 (+3691272us) ts_tablet_manager.cc:1145] Deleting tablet 056ed0fd01824bcea53d58251328898c
0112 20:51:54.524435 (+ 220us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.524946 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3691019}
I20250112 20:51:54.526706 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c2a8f71233d04ca39e49932308e56492
I20250112 20:51:54.527133 455 catalog_manager.cc:6596] ReplaceTablet: tablet c2a8f71233d04ca39e49932308e56492 deleted and replaced by tablet b1b24ca476e34beeac1cd2fd7903db39
I20250112 20:51:54.527791 1185 tablet_replica.cc:331] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.528975 1185 raft_consensus.cc:2238] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.529917 1185 raft_consensus.cc:2267] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.533221 1185 ts_tablet_manager.cc:1905] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.533346 884 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9393e5cc434ec19bb907a6331e9d26 (DEFAULT_TABLE table=default.Table_dc131053ce3845268e1279e76cbf1ddd [id=5c59f232512440c98942e6bd1769f32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.536232 455 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 263b370f96044305a79ebdb79d5e01d1 (table default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4]). Expected version 1 got 0
I20250112 20:51:54.539523 1164 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.539741 1164 ts_tablet_manager.cc:1397] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.003s
I20250112 20:51:54.540514 1164 raft_consensus.cc:357] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.540957 1164 raft_consensus.cc:383] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.541148 1164 raft_consensus.cc:738] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.541514 1164 consensus_queue.cc:260] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.542480 1164 ts_tablet_manager.cc:1428] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:54.543118 1164 tablet_bootstrap.cc:492] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
W20250112 20:51:54.540701 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 78698.2us
I20250112 20:51:54.547668 1185 ts_tablet_manager.cc:1918] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.547972 1185 log.cc:1198] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d80791d8582446b89adf212d3d6507eb
I20250112 20:51:54.548542 1185 ts_tablet_manager.cc:1939] T d80791d8582446b89adf212d3d6507eb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.549121 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1132) took 3624 ms. Trace:
I20250112 20:51:54.549317 1185 rpcz_store.cc:276] 0112 20:51:50.924335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.924383 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:54.527407 (+3603024us) ts_tablet_manager.cc:1145] Deleting tablet d80791d8582446b89adf212d3d6507eb
0112 20:51:54.527420 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.543001 (+ 15581us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.547547 (+ 4546us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.549103 (+ 1556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602841}
I20250112 20:51:54.549822 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ce9397b884b0095dae39ea57f5fd9" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:54.550123 1185 tablet_replica.cc:331] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.550474 1185 raft_consensus.cc:2238] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.550992 1185 raft_consensus.cc:2267] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.552058 1185 ts_tablet_manager.cc:1905] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.557435 710 leader_election.cc:343] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: f425779195594599a3ab8c4dc5bee7f9
I20250112 20:51:54.558679 1164 tablet_bootstrap.cc:654] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.558223 710 leader_election.cc:304] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.559890 3121 raft_consensus.cc:2743] T f425779195594599a3ab8c4dc5bee7f9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.560261 885 tablet_service.cc:1467] Processing CreateTablet for tablet c61749c7999d43b68dc1a44f58125fc8 (DEFAULT_TABLE table=default.Table_c65b1f79389e468fb8793d95d7248128 [id=c1d41ebadc9a4edb9ee8afe0eb739042]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.560992 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a7129e527943589de5a1a28c7e1f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.561218 1185 ts_tablet_manager.cc:1918] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.561460 1185 log.cc:1198] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/d17e6d263014448b96c5d1f03013e6e7
I20250112 20:51:54.561609 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9393e5cc434ec19bb907a6331e9d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.562248 1185 ts_tablet_manager.cc:1939] T d17e6d263014448b96c5d1f03013e6e7 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.562803 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1131) took 3638 ms. Trace:
I20250112 20:51:54.563000 1185 rpcz_store.cc:276] 0112 20:51:50.924081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.924827 (+ 746us) service_pool.cc:225] Handling call
0112 20:51:54.549942 (+3625115us) ts_tablet_manager.cc:1145] Deleting tablet d17e6d263014448b96c5d1f03013e6e7
0112 20:51:54.550059 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.556470 (+ 6411us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.561095 (+ 4625us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.562789 (+ 1694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3624951}
I20250112 20:51:54.563658 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61749c7999d43b68dc1a44f58125fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.563933 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1133) took 3638 ms. Trace:
I20250112 20:51:54.564087 1185 rpcz_store.cc:276] 0112 20:51:50.925606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:50.928829 (+ 3223us) service_pool.cc:225] Handling call
0112 20:51:54.563654 (+3634825us) ts_tablet_manager.cc:1145] Deleting tablet 096ee7d3a66846f9b619d8f2fc046616
0112 20:51:54.563666 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.563889 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3634658}
I20250112 20:51:54.564810 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1134) took 3560 ms. Trace:
I20250112 20:51:54.565845 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 008d69d199d64bdf85da17d8d3bf6264 (table default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]) successfully deleted
I20250112 20:51:54.565973 1185 rpcz_store.cc:276] 0112 20:51:51.004576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.004634 (+ 58us) service_pool.cc:225] Handling call
0112 20:51:54.564639 (+3560005us) ts_tablet_manager.cc:1145] Deleting tablet 8074b00eda8d434faaff4b98ae923ebe
0112 20:51:54.564647 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.564766 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3559793}
W20250112 20:51:54.566182 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 75790518579d496fb0efb12b497fcc75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75790518579d496fb0efb12b497fcc75 already in progress: creating tablet
I20250112 20:51:54.566537 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
I20250112 20:51:54.566781 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1136) took 3207 ms. Trace:
W20250112 20:51:54.566967 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 417ada5111454c54a85cc95db5974bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417ada5111454c54a85cc95db5974bf9 already in progress: creating tablet
W20250112 20:51:54.567441 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 782a195e68124cd391d0394bcb5ec8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 782a195e68124cd391d0394bcb5ec8b2 already in progress: creating tablet
W20250112 20:51:54.567852 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet df292cbdf58b4fcd889fcb63335b37d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df292cbdf58b4fcd889fcb63335b37d9 already in progress: creating tablet
I20250112 20:51:54.566959 1185 rpcz_store.cc:276] 0112 20:51:51.359354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.359396 (+ 42us) service_pool.cc:225] Handling call
0112 20:51:54.566606 (+3207210us) ts_tablet_manager.cc:1145] Deleting tablet d571de1e2b104ba480cdaf2523b54599
0112 20:51:54.566616 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.566741 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207021}
I20250112 20:51:54.568516 3121 raft_consensus.cc:491] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 20:51:54.568637 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 consensus state has no leader
I20250112 20:51:54.568596 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1144) took 3010 ms. Trace:
I20250112 20:51:54.568641 3121 raft_consensus.cc:513] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.568707 1185 rpcz_store.cc:276] 0112 20:51:51.558027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.559358 (+ 1331us) service_pool.cc:225] Handling call
0112 20:51:54.568447 (+3009089us) ts_tablet_manager.cc:1145] Deleting tablet b496462873e44ceab2ec3c95cf63fc5a
0112 20:51:54.568455 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.568583 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3008870}
W20250112 20:51:54.569049 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6e0049fbf174962931c4c3d713535e1 consensus state has no leader
I20250112 20:51:54.569285 3121 leader_election.cc:290] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:54.569622 1185 tablet_replica.cc:331] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.569979 1185 raft_consensus.cc:2238] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.570397 1185 raft_consensus.cc:2267] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.570709 1186 ts_tablet_manager.cc:1918] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.570811 1186 log.cc:1198] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/3994d41a7ecd418a98a3c203add0d4d4
W20250112 20:51:54.570499 710 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.571468 1185 ts_tablet_manager.cc:1905] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.573359 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet c54516970d424f7cb28c72d8fe52b824
I20250112 20:51:54.573877 448 catalog_manager.cc:6596] ReplaceTablet: tablet c54516970d424f7cb28c72d8fe52b824 deleted and replaced by tablet 322bf0831e65498486b323be3b62076c
I20250112 20:51:54.574328 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a87da1ea4054ffc81b9c1c792f0cc5d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:54.574903 710 leader_election.cc:343] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.576694 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a87da1ea4054ffc81b9c1c792f0cc5d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.580837 1164 tablet_bootstrap.cc:492] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.581036 1164 ts_tablet_manager.cc:1397] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20250112 20:51:54.581178 1185 ts_tablet_manager.cc:1918] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 20:51:54.581398 1185 log.cc:1198] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/480d49569886408a8ce781a93bb5d10d
I20250112 20:51:54.581992 1164 raft_consensus.cc:357] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.582170 1185 ts_tablet_manager.cc:1939] T 480d49569886408a8ce781a93bb5d10d P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.582257 1164 raft_consensus.cc:383] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.582404 1164 raft_consensus.cc:738] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.582805 1164 consensus_queue.cc:260] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.583226 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1145) took 2959 ms. Trace:
I20250112 20:51:54.583371 1185 rpcz_store.cc:276] 0112 20:51:51.623381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.623428 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:54.569557 (+2946129us) ts_tablet_manager.cc:1145] Deleting tablet 480d49569886408a8ce781a93bb5d10d
0112 20:51:54.569566 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.576520 (+ 6954us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.581047 (+ 4527us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.583186 (+ 2139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945938}
I20250112 20:51:54.584143 1164 ts_tablet_manager.cc:1428] T 015f54bdae6b4333963f9e085c5189e8 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:54.584297 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1146) took 2911 ms. Trace:
I20250112 20:51:54.584439 1185 rpcz_store.cc:276] 0112 20:51:51.672487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.673708 (+ 1221us) service_pool.cc:225] Handling call
0112 20:51:54.584113 (+2910405us) ts_tablet_manager.cc:1145] Deleting tablet a2ff6b3dbc614785a5db4b4185fe0945
0112 20:51:54.584122 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.584256 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2910239}
I20250112 20:51:54.585122 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1148) took 2886 ms. Trace:
I20250112 20:51:54.585332 1164 tablet_bootstrap.cc:492] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.585289 1185 rpcz_store.cc:276] 0112 20:51:51.699082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.699134 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:54.584978 (+2885844us) ts_tablet_manager.cc:1145] Deleting tablet 75790518579d496fb0efb12b497fcc75
0112 20:51:54.584985 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.585105 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885553}
I20250112 20:51:54.586269 1185 tablet_replica.cc:331] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.587519 1186 ts_tablet_manager.cc:1939] T 3994d41a7ecd418a98a3c203add0d4d4 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.597900 1185 raft_consensus.cc:2238] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.598768 1185 raft_consensus.cc:2267] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.601047 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ff84b2dae9fe4cd5b3bcf625a7d3e829
I20250112 20:51:54.601033 1185 ts_tablet_manager.cc:1905] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.602963 1164 tablet_bootstrap.cc:654] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.619823 3130 raft_consensus.cc:2743] T 691e2c3f72ab4f439678d166b9b31661 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:54.626248 1185 ts_tablet_manager.cc:1918] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.626740 1185 log.cc:1198] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/008d69d199d64bdf85da17d8d3bf6264
I20250112 20:51:54.627782 1185 ts_tablet_manager.cc:1939] T 008d69d199d64bdf85da17d8d3bf6264 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.629099 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1147) took 2930 ms. Trace:
I20250112 20:51:54.629386 1185 rpcz_store.cc:276] 0112 20:51:51.698440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.699410 (+ 970us) service_pool.cc:225] Handling call
0112 20:51:54.586121 (+2886711us) ts_tablet_manager.cc:1145] Deleting tablet 008d69d199d64bdf85da17d8d3bf6264
0112 20:51:54.586129 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.609870 (+ 23741us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.625898 (+ 16028us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.629030 (+ 3132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886563}
I20250112 20:51:54.630961 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1152) took 2927 ms. Trace:
I20250112 20:51:54.631268 1185 rpcz_store.cc:276] 0112 20:51:51.703936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.704187 (+ 251us) service_pool.cc:225] Handling call
0112 20:51:54.630644 (+2926457us) ts_tablet_manager.cc:1145] Deleting tablet 669e1cfd407e4d87a63e6832d9c6fa1f
0112 20:51:54.630697 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.630907 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2926247}
I20250112 20:51:54.633038 1185 tablet_replica.cc:331] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.633383 1185 raft_consensus.cc:2238] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.633700 1185 raft_consensus.cc:2267] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.634800 1164 tablet_bootstrap.cc:492] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.635049 1164 ts_tablet_manager.cc:1397] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.004s
I20250112 20:51:54.636411 3114 raft_consensus.cc:491] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.636514 3114 raft_consensus.cc:513] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.636356 1164 raft_consensus.cc:357] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.636687 1164 raft_consensus.cc:383] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.636767 1164 raft_consensus.cc:738] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
W20250112 20:51:54.637213 710 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 868ce9397b884b0095dae39ea57f5fd9
I20250112 20:51:54.637099 1164 consensus_queue.cc:260] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.637333 710 leader_election.cc:304] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.638037 1164 ts_tablet_manager.cc:1428] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:54.638428 1164 tablet_bootstrap.cc:492] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.638458 3539 tablet.cc:1721] T 263b370f96044305a79ebdb79d5e01d1 P 30ba399c7a6749e0acd075d93e4cc4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:54.638585 3470 raft_consensus.cc:2743] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.640507 1185 ts_tablet_manager.cc:1905] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.568364 440 catalog_manager.cc:4852] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): delete failed for tablet 91243c9c6deb447e8246f7abe2691f55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91243c9c6deb447e8246f7abe2691f55 already in progress: creating tablet
W20250112 20:51:54.641511 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36593, user_credentials={real_user=slave}} blocked reactor thread for 73188.9us
I20250112 20:51:54.642302 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 22f6b2c4ea5f4167896c78b1c64cbb62 (table default.Table_cc55237c53344dfbbf00a7b33b6962e9 [id=992620c02c1b401a8936586a1ef0d291]) successfully deleted
I20250112 20:51:54.642477 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 9570269368e14b1abeaffdcc741c6f3a (table default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]) successfully deleted
W20250112 20:51:54.642656 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8bc545822e98460e815df4e6cad455b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc545822e98460e815df4e6cad455b4 already in progress: creating tablet
W20250112 20:51:54.642839 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 5065a94bb13c419e9d9b8eaca72f5052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5065a94bb13c419e9d9b8eaca72f5052 already in progress: creating tablet
W20250112 20:51:54.642982 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 6181b166b5cb429da3ebba23cf267942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6181b166b5cb429da3ebba23cf267942 already in progress: creating tablet
I20250112 20:51:54.637096 3114 leader_election.cc:290] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:54.643168 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet db9178cf782d4bd28eba90cc03796885 (table default.Table_e55e24b020a64e9ebaf27b249838c000 [id=16c454ac411b44a3b5d282369bb13e42]) successfully deleted
W20250112 20:51:54.643321 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet f7b722b0eb5944a4b344a3833905a2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7b722b0eb5944a4b344a3833905a2b0 already in progress: creating tablet
W20250112 20:51:54.643471 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 886690f24fa540a6b496c926bc7a2b07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886690f24fa540a6b496c926bc7a2b07 already in progress: creating tablet
W20250112 20:51:54.643622 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 056ed0fd01824bcea53d58251328898c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 056ed0fd01824bcea53d58251328898c already in progress: creating tablet
I20250112 20:51:54.643851 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d80791d8582446b89adf212d3d6507eb (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
I20250112 20:51:54.643990 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet d17e6d263014448b96c5d1f03013e6e7 (table default.Table_a1b5b3029a0f4295904392987db19062 [id=04bd0f25c5db41039b53f79ce61124ae]) successfully deleted
W20250112 20:51:54.644151 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 096ee7d3a66846f9b619d8f2fc046616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 096ee7d3a66846f9b619d8f2fc046616 already in progress: creating tablet
W20250112 20:51:54.644320 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 8074b00eda8d434faaff4b98ae923ebe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8074b00eda8d434faaff4b98ae923ebe already in progress: creating tablet
W20250112 20:51:54.644472 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet d571de1e2b104ba480cdaf2523b54599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d571de1e2b104ba480cdaf2523b54599 already in progress: creating tablet
W20250112 20:51:54.644620 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet b496462873e44ceab2ec3c95cf63fc5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b496462873e44ceab2ec3c95cf63fc5a already in progress: creating tablet
I20250112 20:51:54.646679 1164 tablet_bootstrap.cc:654] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:54.649379 710 leader_election.cc:343] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.649606 710 leader_election.cc:304] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.650481 3544 raft_consensus.cc:2743] T 0a87da1ea4054ffc81b9c1c792f0cc5d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 20:51:54.651278 710 consensus_peers.cc:487] T 691e2c3f72ab4f439678d166b9b31661 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 691e2c3f72ab4f439678d166b9b31661. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:54.651515 1178 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:54.651789 1178 ts_tablet_manager.cc:1397] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.354s user 0.011s sys 0.006s
I20250112 20:51:54.652076 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37e9e95c6b745ccaba5446b04abcc9d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.652506 1178 raft_consensus.cc:357] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.652675 1178 raft_consensus.cc:383] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.652734 1178 raft_consensus.cc:738] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.652968 1178 consensus_queue.cc:260] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.654613 1178 ts_tablet_manager.cc:1428] T 05550152671e4e28829b5b102b59f715 P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:54.662838 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37e9e95c6b745ccaba5446b04abcc9d" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:54.664835 983 consensus_peers.cc:487] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): Couldn't send request to peer aefe1db8b19244d891b6bcd8971e9b17. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:54.666833 1186 tablet_replica.cc:331] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.667074 1186 raft_consensus.cc:2238] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.667294 1186 raft_consensus.cc:2267] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.668321 1186 ts_tablet_manager.cc:1905] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.670429 1163 tablet_bootstrap.cc:492] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.671911 1163 ts_tablet_manager.cc:1397] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.309s user 0.006s sys 0.000s
I20250112 20:51:54.672943 1163 raft_consensus.cc:357] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.673311 1178 tablet_bootstrap.cc:492] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:54.673178 1185 ts_tablet_manager.cc:1918] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.673756 1185 log.cc:1198] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/24a5162a25ce47519a091742f3d5ca31
I20250112 20:51:54.674340 1163 raft_consensus.cc:383] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.674607 1163 raft_consensus.cc:738] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.675092 1178 tablet_bootstrap.cc:654] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.675014 1163 consensus_queue.cc:260] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.676312 1185 ts_tablet_manager.cc:1939] T 24a5162a25ce47519a091742f3d5ca31 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.677937 1163 ts_tablet_manager.cc:1428] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 20:51:54.681645 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1153) took 2977 ms. Trace:
I20250112 20:51:54.682114 1185 rpcz_store.cc:276] 0112 20:51:51.704198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.704374 (+ 176us) service_pool.cc:225] Handling call
0112 20:51:54.632968 (+2928594us) ts_tablet_manager.cc:1145] Deleting tablet 24a5162a25ce47519a091742f3d5ca31
0112 20:51:54.632977 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.652638 (+ 19661us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.672528 (+ 19890us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.681497 (+ 8969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2928419}
I20250112 20:51:54.684231 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1156) took 2806 ms. Trace:
I20250112 20:51:54.644755 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 480d49569886408a8ce781a93bb5d10d (table default.Table_cb2b5abd46bc4877887604a94159e418 [id=d46a20ad8b99492395771d94f5dc60fb]) successfully deleted
W20250112 20:51:54.685117 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet a2ff6b3dbc614785a5db4b4185fe0945 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff6b3dbc614785a5db4b4185fe0945 already in progress: creating tablet
W20250112 20:51:54.685451 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 75790518579d496fb0efb12b497fcc75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75790518579d496fb0efb12b497fcc75 already in progress: creating tablet
I20250112 20:51:54.684828 1185 rpcz_store.cc:276] 0112 20:51:51.877629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.877675 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:54.683547 (+2805872us) ts_tablet_manager.cc:1145] Deleting tablet 417ada5111454c54a85cc95db5974bf9
0112 20:51:54.683558 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.684098 (+ 540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805571}
I20250112 20:51:54.685885 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 008d69d199d64bdf85da17d8d3bf6264 (table default.Table_292dce92952a4a3ba27a7bb1ed129e3e [id=122e3443c1184f369612baf1236c24a0]) successfully deleted
W20250112 20:51:54.686843 440 catalog_manager.cc:4839] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 669e1cfd407e4d87a63e6832d9c6fa1f because the tablet was not found. No further retry: Not found: Tablet not found: 669e1cfd407e4d87a63e6832d9c6fa1f
I20250112 20:51:54.687465 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 24a5162a25ce47519a091742f3d5ca31 (table default.Table_7f122343e2d74bf7b3eefea9de3ff725 [id=02dcec3eeef04e78a9c707989a8578bc]) successfully deleted
I20250112 20:51:54.687059 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1157) took 2807 ms. Trace:
W20250112 20:51:54.688028 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 417ada5111454c54a85cc95db5974bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417ada5111454c54a85cc95db5974bf9 already in progress: creating tablet
I20250112 20:51:54.687994 1185 rpcz_store.cc:276] 0112 20:51:51.879578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.879630 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:54.686299 (+2806669us) ts_tablet_manager.cc:1145] Deleting tablet 782a195e68124cd391d0394bcb5ec8b2
0112 20:51:54.686407 (+ 108us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.686909 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2806393}
I20250112 20:51:54.689635 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 868ce9397b884b0095dae39ea57f5fd9 (table default.Table_455466a0838742f3a2cc35cdab729433 [id=fada45a1d9a64ff4871c139b8767f08e]) successfully deleted
W20250112 20:51:54.690114 440 catalog_manager.cc:4839] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c because the tablet was not found. No further retry: Not found: Tablet not found: 8dafaa3a1a3d4d8bb8aa0448a61e237c
I20250112 20:51:54.690626 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1158) took 2769 ms. Trace:
I20250112 20:51:54.693354 3513 raft_consensus.cc:491] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.693500 3513 raft_consensus.cc:513] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.694696 3513 leader_election.cc:290] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.692523 1185 rpcz_store.cc:276] 0112 20:51:51.920022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.920077 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:54.689447 (+2769370us) ts_tablet_manager.cc:1145] Deleting tablet df292cbdf58b4fcd889fcb63335b37d9
0112 20:51:54.689457 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.689873 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2769001}
W20250112 20:51:54.695485 983 leader_election.cc:343] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b30a34c1abcb4f72a7f4f8a8f5be3720
I20250112 20:51:54.696452 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1159) took 2771 ms. Trace:
I20250112 20:51:54.696892 1185 rpcz_store.cc:276] 0112 20:51:51.925117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.925163 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:54.695976 (+2770813us) ts_tablet_manager.cc:1145] Deleting tablet 91243c9c6deb447e8246f7abe2691f55
0112 20:51:54.695986 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.696434 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2770499}
I20250112 20:51:54.698503 1185 tablet_replica.cc:331] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.699997 1185 raft_consensus.cc:2238] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.700907 1185 raft_consensus.cc:2267] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:51:54.701494 710 leader_election.cc:343] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.703330 1186 ts_tablet_manager.cc:1918] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:54.704027 1185 ts_tablet_manager.cc:1905] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.705555 1186 log.cc:1198] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/235faeb3c35d46d2882ffcd8cb63ddb8
I20250112 20:51:54.706354 1186 ts_tablet_manager.cc:1939] T 235faeb3c35d46d2882ffcd8cb63ddb8 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.717789 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61749c7999d43b68dc1a44f58125fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.719412 3391 raft_consensus.cc:491] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.719605 3391 raft_consensus.cc:513] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.720290 3391 leader_election.cc:290] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
W20250112 20:51:54.728991 3258 log.cc:926] Time spent T 00000000000000000000000000000000 P b985fd656ce44131bc91ccfeb79587c5: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250112 20:51:54.729132 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9020192a4d34e8eb8590c41d545efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.731598 3513 raft_consensus.cc:491] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.731729 3513 raft_consensus.cc:513] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.732299 3513 leader_election.cc:290] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.732756 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720aa779aada46d5a888d5011804a6a8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.732980 901 raft_consensus.cc:2463] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:54.733404 983 leader_election.cc:304] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters:
I20250112 20:51:54.733695 3513 raft_consensus.cc:2798] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.733784 3513 raft_consensus.cc:491] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.733811 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023ab1730f1a4abbaca73018d518b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.733852 3513 raft_consensus.cc:3054] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.736109 3513 raft_consensus.cc:513] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.737949 3513 leader_election.cc:290] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.738620 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720aa779aada46d5a888d5011804a6a8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3"
I20250112 20:51:54.738785 913 raft_consensus.cc:3054] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.742758 1163 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.744112 1163 tablet_bootstrap.cc:654] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.748401 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30a34c1abcb4f72a7f4f8a8f5be3720" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.748610 766 raft_consensus.cc:2463] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:54.748672 1185 ts_tablet_manager.cc:1918] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.749239 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720aa779aada46d5a888d5011804a6a8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
W20250112 20:51:54.748771 710 leader_election.cc:343] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.749472 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720aa779aada46d5a888d5011804a6a8" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:54.749313 1185 log.cc:1198] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/b37b00c49f234dc89d63f644febc23fb
I20250112 20:51:54.749544 710 leader_election.cc:304] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.749387 763 raft_consensus.cc:2463] T 720aa779aada46d5a888d5011804a6a8 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
I20250112 20:51:54.750349 3391 raft_consensus.cc:2743] T b37e9e95c6b745ccaba5446b04abcc9d P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.750859 1185 ts_tablet_manager.cc:1939] T b37b00c49f234dc89d63f644febc23fb P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.752291 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1160) took 2812 ms. Trace:
W20250112 20:51:54.754539 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 64422.8us
I20250112 20:51:54.755491 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 3994d41a7ecd418a98a3c203add0d4d4 (table default.Table_7038f1cfb0f944378fdf6ab8daa15d60 [id=e2d5f4685f49452eb6574236b4fb0c52]) successfully deleted
I20250112 20:51:54.753065 1185 rpcz_store.cc:276] 0112 20:51:51.939868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.939947 (+ 79us) service_pool.cc:225] Handling call
0112 20:51:54.698241 (+2758294us) ts_tablet_manager.cc:1145] Deleting tablet b37b00c49f234dc89d63f644febc23fb
0112 20:51:54.698351 (+ 110us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.722302 (+ 23951us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.748259 (+ 25957us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.752154 (+ 3895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2757969}
I20250112 20:51:54.755959 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 235faeb3c35d46d2882ffcd8cb63ddb8 (table default.Table_c5a2d4a4380c473bb751bd109b159990 [id=2c1e80f117244d4a91c59578fbc1245f]) successfully deleted
W20250112 20:51:54.756207 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 02264625e0d9441882ecefcc8cc36227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02264625e0d9441882ecefcc8cc36227 already in progress: creating tablet
W20250112 20:51:54.756476 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 42c88852d29545c7a937ca389929d2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42c88852d29545c7a937ca389929d2b1 already in progress: creating tablet
W20250112 20:51:54.756704 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 023ab1730f1a4abbaca73018d518b1a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 023ab1730f1a4abbaca73018d518b1a3 already in progress: creating tablet
I20250112 20:51:54.757934 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:54.758246 1039 raft_consensus.cc:2463] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 1.
I20250112 20:51:54.757483 1185 tablet_replica.cc:331] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.759521 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_917e8a15aa6b4cc28455521407819e42" } modify_external_catalogs: true
I20250112 20:51:54.759685 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_917e8a15aa6b4cc28455521407819e42" } modify_external_catalogs: true
I20250112 20:51:54.760118 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fdbdfd9f44f6454c9839477f1f437558" } modify_external_catalogs: true
I20250112 20:51:54.760241 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_fdbdfd9f44f6454c9839477f1f437558" } modify_external_catalogs: true
I20250112 20:51:54.760659 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dae7a8c6178b4a5aab35a17c2abd509f" } modify_external_catalogs: true
I20250112 20:51:54.760982 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_e362fd930fa843b4bbbba8c04292066b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.760571 1185 raft_consensus.cc:2238] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 20:51:54.761538 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e362fd930fa843b4bbbba8c04292066b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.761817 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_669829b96b0a4f1fab410ba1e9b1e5d6" } modify_external_catalogs: true
I20250112 20:51:54.761914 1163 tablet_bootstrap.cc:492] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.762037 1163 ts_tablet_manager.cc:1397] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250112 20:51:54.762147 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_4b6bc53207aa4a2387d9ecb440780709"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.762555 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b6bc53207aa4a2387d9ecb440780709 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.762832 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1f02207c5f8b4c38bd675da4c908ebf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.762879 1163 raft_consensus.cc:357] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.763157 1163 raft_consensus.cc:383] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 20:51:54.763237 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f02207c5f8b4c38bd675da4c908ebf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.763258 1163 raft_consensus.cc:738] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.763518 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_45b4a66f478745238d02cdf278799afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.763615 1163 consensus_queue.cc:260] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:54.763897 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45b4a66f478745238d02cdf278799afd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.764191 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_13e489ec495e4045afede54e22d5d79d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:51:54.764282 1163 ts_tablet_manager.cc:1428] T 3994d41a7ecd418a98a3c203add0d4d4 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250112 20:51:54.764565 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13e489ec495e4045afede54e22d5d79d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.764806 1163 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.765183 1186 tablet_replica.cc:331] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.765375 1186 raft_consensus.cc:2238] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.765511 1186 raft_consensus.cc:2267] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.765992 1186 ts_tablet_manager.cc:1905] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.766075 1163 tablet_bootstrap.cc:654] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.770668 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_dae7a8c6178b4a5aab35a17c2abd509f" } modify_external_catalogs: true
I20250112 20:51:54.775827 1185 raft_consensus.cc:2267] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.778285 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_669829b96b0a4f1fab410ba1e9b1e5d6" } modify_external_catalogs: true
I20250112 20:51:54.779019 2447 ts_tablet_manager.cc:1918] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.778976 983 leader_election.cc:304] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.779260 2447 log.cc:1198] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d80791d8582446b89adf212d3d6507eb
I20250112 20:51:54.779569 2447 ts_tablet_manager.cc:1939] T d80791d8582446b89adf212d3d6507eb P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.779608 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a8f71233d04ca39e49932308e56492 with delete_type TABLET_DATA_DELETED (replaced by tablet b1b24ca476e34beeac1cd2fd7903db39) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.779826 3513 raft_consensus.cc:2798] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.779891 3513 raft_consensus.cc:491] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.779917 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1099) took 2791 ms. Trace:
I20250112 20:51:54.779963 3513 raft_consensus.cc:3054] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.779947 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a8f71233d04ca39e49932308e56492 with delete_type TABLET_DATA_DELETED (replaced by tablet b1b24ca476e34beeac1cd2fd7903db39) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.780892 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet c54516970d424f7cb28c72d8fe52b824 with delete_type TABLET_DATA_DELETED (replaced by tablet 322bf0831e65498486b323be3b62076c) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.782213 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a8f71233d04ca39e49932308e56492 with delete_type TABLET_DATA_DELETED (replaced by tablet b1b24ca476e34beeac1cd2fd7903db39) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.782480 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c54516970d424f7cb28c72d8fe52b824 with delete_type TABLET_DATA_DELETED (replaced by tablet 322bf0831e65498486b323be3b62076c) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.782773 3513 raft_consensus.cc:513] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:54.784016 983 leader_election.cc:343] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: b30a34c1abcb4f72a7f4f8a8f5be3720
I20250112 20:51:54.784181 3513 leader_election.cc:290] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.779974 2447 rpcz_store.cc:276] 0112 20:51:51.988945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.990767 (+ 1822us) service_pool.cc:225] Handling call
0112 20:51:54.443182 (+2452415us) ts_tablet_manager.cc:1145] Deleting tablet d80791d8582446b89adf212d3d6507eb
0112 20:51:54.443190 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.716752 (+273562us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.778948 (+ 62196us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.779904 (+ 956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387403}
I20250112 20:51:54.785426 2447 tablet_replica.cc:331] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.785732 2447 raft_consensus.cc:2238] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.786317 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.786010 2447 raft_consensus.cc:2267] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.786394 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:54.788236 913 raft_consensus.cc:2463] T 720aa779aada46d5a888d5011804a6a8 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:54.789022 2447 ts_tablet_manager.cc:1905] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.789215 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ff84b2dae9fe4cd5b3bcf625a7d3e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.790272 1163 tablet_bootstrap.cc:492] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.790423 1163 ts_tablet_manager.cc:1397] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250112 20:51:54.790740 1185 ts_tablet_manager.cc:1905] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.791651 1163 raft_consensus.cc:357] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.791857 1163 raft_consensus.cc:383] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.791932 1163 raft_consensus.cc:738] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.792145 1163 consensus_queue.cc:260] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.795301 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet d80791d8582446b89adf212d3d6507eb (table default.Table_b5e7418fef2d44709ff22f9961666ee3 [id=abd0f618fc7b449db908c4865a4538d6]) successfully deleted
I20250112 20:51:54.795539 983 leader_election.cc:304] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3; no voters: aefe1db8b19244d891b6bcd8971e9b17
I20250112 20:51:54.795759 3513 raft_consensus.cc:2798] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.795957 3513 raft_consensus.cc:695] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
W20250112 20:51:54.796002 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 782a195e68124cd391d0394bcb5ec8b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 782a195e68124cd391d0394bcb5ec8b2 already in progress: creating tablet
W20250112 20:51:54.796200 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet df292cbdf58b4fcd889fcb63335b37d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df292cbdf58b4fcd889fcb63335b37d9 already in progress: creating tablet
I20250112 20:51:54.796191 3513 consensus_queue.cc:237] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
W20250112 20:51:54.796370 440 catalog_manager.cc:4852] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): delete failed for tablet 91243c9c6deb447e8246f7abe2691f55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91243c9c6deb447e8246f7abe2691f55 already in progress: creating tablet
I20250112 20:51:54.796518 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet b37b00c49f234dc89d63f644febc23fb (table default.Table_e907fd649a434fe9b3a51b2619964cb3 [id=8997a20696974e4fb75b4f18903d7def]) successfully deleted
W20250112 20:51:54.796694 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet d8ea5b6d791d4891a1bb008879eba941 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ea5b6d791d4891a1bb008879eba941 already in progress: creating tablet
I20250112 20:51:54.795766 1163 ts_tablet_manager.cc:1428] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 20:51:54.797060 1163 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.798381 1163 tablet_bootstrap.cc:654] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
W20250112 20:51:54.803721 440 catalog_manager.cc:4852] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): delete failed for tablet 3b8649f8404240df8b750f431a835516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8649f8404240df8b750f431a835516 already in progress: creating tablet
I20250112 20:51:54.806196 710 leader_election.cc:304] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters:
I20250112 20:51:54.808326 3265 raft_consensus.cc:2798] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:54.808425 3265 raft_consensus.cc:491] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.808521 3265 raft_consensus.cc:3054] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:54.821408 763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30a34c1abcb4f72a7f4f8a8f5be3720" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:54.821585 763 raft_consensus.cc:3054] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.823935 763 raft_consensus.cc:2463] T b30a34c1abcb4f72a7f4f8a8f5be3720 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
W20250112 20:51:54.823868 710 leader_election.cc:343] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 615d75bc22df45cc876788a266e983cc
I20250112 20:51:54.825018 1186 ts_tablet_manager.cc:1918] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.826234 1163 tablet_bootstrap.cc:492] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.826406 1163 ts_tablet_manager.cc:1397] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20250112 20:51:54.827239 1163 raft_consensus.cc:357] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.827458 1163 raft_consensus.cc:383] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.827531 1163 raft_consensus.cc:738] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.827755 1163 consensus_queue.cc:260] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.828271 1163 ts_tablet_manager.cc:1428] T 05550152671e4e28829b5b102b59f715 P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:54.828539 1163 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.830368 1186 log.cc:1198] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/43d341f6ac6848d9a8e60f689dd24a66
I20250112 20:51:54.830755 1186 ts_tablet_manager.cc:1939] T 43d341f6ac6848d9a8e60f689dd24a66 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.831087 1163 tablet_bootstrap.cc:654] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.831578 1186 tablet_replica.cc:331] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: stopping tablet replica
I20250112 20:51:54.837252 983 leader_election.cc:304] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.837311 440 catalog_manager.cc:4872] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): tablet 43d341f6ac6848d9a8e60f689dd24a66 (table default.Table_9b86e02c3663476d9f4b850bf5982df8 [id=de08a382022148708327b2383827d8ae]) successfully deleted
I20250112 20:51:54.837651 3513 raft_consensus.cc:2798] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.837831 3513 raft_consensus.cc:695] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:54.838040 3513 consensus_queue.cc:237] T b30a34c1abcb4f72a7f4f8a8f5be3720 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.845126 1186 raft_consensus.cc:2238] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:54.845453 1186 raft_consensus.cc:2267] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.845984 1186 ts_tablet_manager.cc:1905] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:54.854101 1178 tablet_bootstrap.cc:492] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:54.855803 1178 ts_tablet_manager.cc:1397] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.010s
I20250112 20:51:54.860374 1178 raft_consensus.cc:357] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.860651 1178 raft_consensus.cc:383] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.860746 1178 raft_consensus.cc:738] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.861008 1178 consensus_queue.cc:260] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.866585 2447 ts_tablet_manager.cc:1918] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:54.866647 1185 ts_tablet_manager.cc:1918] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:54.867417 1185 log.cc:1198] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/65784493c6ef417993b1f6371cab47d6
I20250112 20:51:54.867362 2447 log.cc:1198] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/9570269368e14b1abeaffdcc741c6f3a
I20250112 20:51:54.868786 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_18e393adcff3469cbc4acfc94fca9752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.869359 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e393adcff3469cbc4acfc94fca9752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.870142 1164 tablet_bootstrap.cc:492] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.870343 1164 ts_tablet_manager.cc:1397] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.232s user 0.007s sys 0.027s
I20250112 20:51:54.871120 1164 raft_consensus.cc:357] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.871346 1164 raft_consensus.cc:383] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.871462 1164 raft_consensus.cc:738] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.871760 1178 ts_tablet_manager.cc:1428] T 2de03e3114d145aeb81590dbae2c548e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 20:51:54.871731 1164 consensus_queue.cc:260] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.872121 1178 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:54.872289 1164 ts_tablet_manager.cc:1428] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:54.872676 1164 tablet_bootstrap.cc:492] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.873505 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55686:
table { table_name: "default.Table_0d4b1fc5d57d4e14b2ef18ab874b4391" } new_table_name: "default.Table_7a7c674f24a74321a70f1d357d71cffb" modify_external_catalogs: true
I20250112 20:51:54.874667 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55686:
name: "default.Table_1fd914125f6c4872a0ab7d267bc88685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 20:51:54.875130 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd914125f6c4872a0ab7d267bc88685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:54.874610 1164 tablet_bootstrap.cc:654] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.878906 1185 ts_tablet_manager.cc:1939] T 65784493c6ef417993b1f6371cab47d6 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:54.857316 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c54516970d424f7cb28c72d8fe52b824 with delete_type TABLET_DATA_DELETED (replaced by tablet 322bf0831e65498486b323be3b62076c) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.856743 3563 tablet.cc:1721] T 263b370f96044305a79ebdb79d5e01d1 P 87ae45957c64469d96dc92d606b5b4b3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 20:51:54.881994 449 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 05550152671e4e28829b5b102b59f715 (table default.Table_8d7496dac6374e5186e874f341b74977 [id=4b364d74bdc149a09a4765f2c125d35c]). Expected version 2 got 0
I20250112 20:51:54.882688 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 6921185b545244978f8fdeafd1fb6c5e from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.884122 1163 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.884305 1163 ts_tablet_manager.cc:1397] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.004s
I20250112 20:51:54.884896 453 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 05550152671e4e28829b5b102b59f715 (table default.Table_8d7496dac6374e5186e874f341b74977 [id=4b364d74bdc149a09a4765f2c125d35c]). Expected version 2 got 0
I20250112 20:51:54.885069 1163 raft_consensus.cc:357] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.885278 1163 raft_consensus.cc:383] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.885360 1163 raft_consensus.cc:738] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.885619 1163 consensus_queue.cc:260] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.886231 1163 ts_tablet_manager.cc:1428] T b37e9e95c6b745ccaba5446b04abcc9d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:54.886556 2447 ts_tablet_manager.cc:1939] T 9570269368e14b1abeaffdcc741c6f3a P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:54.887321 1163 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:54.887876 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9570269368e14b1abeaffdcc741c6f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:35 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.888101 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 868ce9397b884b0095dae39ea57f5fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:38 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.888372 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f425779195594599a3ab8c4dc5bee7f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b009608f514353be8a58408b4abdea) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.889518 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b4b312c558437892388344767f0ed4 with delete_type TABLET_DATA_DELETED (replaced by tablet 05dc96770e484cfc8ca8388114ddbcf4) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.894883 1039 raft_consensus.cc:3054] T 68b4b312c558437892388344767f0ed4 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.898674 3121 consensus_queue.cc:1035] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.900476 1163 tablet_bootstrap.cc:654] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.908111 1178 tablet_bootstrap.cc:654] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.910037 1164 tablet_bootstrap.cc:492] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:54.910180 1164 ts_tablet_manager.cc:1397] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250112 20:51:54.910806 1164 raft_consensus.cc:357] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.911005 1164 raft_consensus.cc:383] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.911072 1164 raft_consensus.cc:738] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.911363 1164 consensus_queue.cc:260] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.912309 1164 ts_tablet_manager.cc:1428] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 20:51:54.912714 1164 tablet_bootstrap.cc:492] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:54.913110 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:54.916177 1164 tablet_bootstrap.cc:654] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.918335 1163 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3: No bootstrap required, opened a new log
I20250112 20:51:54.919458 1163 ts_tablet_manager.cc:1397] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20250112 20:51:54.921144 1163 raft_consensus.cc:357] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.921628 1163 raft_consensus.cc:383] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.921784 1163 raft_consensus.cc:738] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.922304 1163 consensus_queue.cc:260] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.923475 3513 raft_consensus.cc:491] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.923575 3513 raft_consensus.cc:513] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.924099 3513 leader_election.cc:290] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.924695 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ce9397b884b0095dae39ea57f5fd9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17" is_pre_election: true
I20250112 20:51:54.924896 759 raft_consensus.cc:2463] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 0.
W20250112 20:51:54.925259 983 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 868ce9397b884b0095dae39ea57f5fd9
I20250112 20:51:54.925382 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 05550152671e4e28829b5b102b59f715 (table default.Table_8d7496dac6374e5186e874f341b74977 [id=4b364d74bdc149a09a4765f2c125d35c]). Expected version 2 got 0
I20250112 20:51:54.925676 453 catalog_manager.cc:5526] T 720aa779aada46d5a888d5011804a6a8 P 30ba399c7a6749e0acd075d93e4cc4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:54.926110 453 catalog_manager.cc:5618] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593) does not have the latest schema for tablet 720aa779aada46d5a888d5011804a6a8 (table default.Table_55404e958b16400f9a4f336dc5f3cd32 [id=ff62e1f5b1aa45408d49878b0c050510]). Expected version 1 got 0
I20250112 20:51:54.927363 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1100) took 2938 ms. Trace:
I20250112 20:51:54.927439 2447 rpcz_store.cc:276] 0112 20:51:51.989049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.990944 (+ 1895us) service_pool.cc:225] Handling call
0112 20:51:54.785346 (+2794402us) ts_tablet_manager.cc:1145] Deleting tablet 9570269368e14b1abeaffdcc741c6f3a
0112 20:51:54.785356 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.829902 (+ 44546us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.866162 (+ 36260us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.927345 (+ 61183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2729497}
I20250112 20:51:54.927707 2447 tablet_replica.cc:331] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:54.928622 983 leader_election.cc:304] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.929415 3513 raft_consensus.cc:2798] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:54.929490 3513 raft_consensus.cc:491] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:54.929548 3513 raft_consensus.cc:3054] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.933925 3513 raft_consensus.cc:513] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.935084 1186 ts_tablet_manager.cc:1918] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 20:51:54.935411 3121 raft_consensus.cc:491] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.935552 3121 raft_consensus.cc:513] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:54.935894 983 leader_election.cc:343] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 868ce9397b884b0095dae39ea57f5fd9
I20250112 20:51:54.936137 3121 leader_election.cc:290] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:54.936426 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868ce9397b884b0095dae39ea57f5fd9" candidate_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe1db8b19244d891b6bcd8971e9b17"
I20250112 20:51:54.936586 759 raft_consensus.cc:3054] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.936586 3513 leader_election.cc:290] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Requested vote from peers aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.936728 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d521e6c8cd461887f93f395c27b89e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:54.937211 710 leader_election.cc:343] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.936479 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d521e6c8cd461887f93f395c27b89e" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:54.937722 710 leader_election.cc:343] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.937814 710 leader_election.cc:304] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.938254 1186 log.cc:1198] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-1/wal/wals/c2a8f71233d04ca39e49932308e56492
I20250112 20:51:54.939461 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 13be813db4f54393bb2d9e80726e3472 (DEFAULT_TABLE table=default.Table_089393cb2b27493a9603c9a1ec1c8959 [id=bdcd915a06b44fd7a159647763ed5ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.939589 751 tablet_service.cc:1467] Processing CreateTablet for tablet 13be813db4f54393bb2d9e80726e3472 (DEFAULT_TABLE table=default.Table_089393cb2b27493a9603c9a1ec1c8959 [id=bdcd915a06b44fd7a159647763ed5ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.939716 3121 raft_consensus.cc:2743] T f8d521e6c8cd461887f93f395c27b89e P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.939817 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be813db4f54393bb2d9e80726e3472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.939982 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be813db4f54393bb2d9e80726e3472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.940251 3121 raft_consensus.cc:491] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:54.940358 3121 raft_consensus.cc:513] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.940888 3121 leader_election.cc:290] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:54.943709 891 tablet_service.cc:1467] Processing CreateTablet for tablet 52476e61dc3b4e20ace9d843f0350c6e (DEFAULT_TABLE table=default.Table_815f2be38a9241d6a8a478e8babdf813 [id=634d926e9f66492aa240438f9b2186ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.943782 749 tablet_service.cc:1467] Processing CreateTablet for tablet 52476e61dc3b4e20ace9d843f0350c6e (DEFAULT_TABLE table=default.Table_815f2be38a9241d6a8a478e8babdf813 [id=634d926e9f66492aa240438f9b2186ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.944056 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52476e61dc3b4e20ace9d843f0350c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.944133 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52476e61dc3b4e20ace9d843f0350c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.945139 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 52476e61dc3b4e20ace9d843f0350c6e (DEFAULT_TABLE table=default.Table_815f2be38a9241d6a8a478e8babdf813 [id=634d926e9f66492aa240438f9b2186ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.945418 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52476e61dc3b4e20ace9d843f0350c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.947320 440 catalog_manager.cc:4872] TS 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): tablet 9570269368e14b1abeaffdcc741c6f3a (table default.Table_bb77de2d74804434a45865840ea7eea0 [id=9703a82e9d0a4aa9928116611bac3830]) successfully deleted
I20250112 20:51:54.947772 882 tablet_service.cc:1467] Processing CreateTablet for tablet 13be813db4f54393bb2d9e80726e3472 (DEFAULT_TABLE table=default.Table_089393cb2b27493a9603c9a1ec1c8959 [id=bdcd915a06b44fd7a159647763ed5ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:54.948102 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13be813db4f54393bb2d9e80726e3472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.949561 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb6a5146c44edea076a10da58ea1a7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:54.950148 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb6a5146c44edea076a10da58ea1a7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:54.953022 710 leader_election.cc:343] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:54.953171 710 leader_election.cc:343] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:54.953229 710 leader_election.cc:304] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.953300 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39271
I20250112 20:51:54.953415 3121 raft_consensus.cc:2743] T f0fb6a5146c44edea076a10da58ea1a7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:54.957552 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1161) took 3015 ms. Trace:
I20250112 20:51:54.957638 1185 rpcz_store.cc:276] 0112 20:51:51.942333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.942380 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:54.756231 (+2813851us) ts_tablet_manager.cc:1145] Deleting tablet 65784493c6ef417993b1f6371cab47d6
0112 20:51:54.757313 (+ 1082us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.830045 (+ 72732us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.866304 (+ 36259us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:54.957539 (+ 91235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2813644}
I20250112 20:51:54.958065 1185 tablet_replica.cc:331] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:54.958964 440 catalog_manager.cc:4872] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585): tablet 65784493c6ef417993b1f6371cab47d6 (table default.Table_893cdd72c53441f3b51cd4eeb6cda9f6 [id=4130b8a93c4645839e4fad41a2127529]) successfully deleted
I20250112 20:51:54.959116 3470 consensus_queue.cc:1035] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:54.961689 1186 ts_tablet_manager.cc:1939] T c2a8f71233d04ca39e49932308e56492 P 87ae45957c64469d96dc92d606b5b4b3: Deleting consensus metadata
I20250112 20:51:54.963128 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet ed7885fed610416792ba4c6df2a80f12
W20250112 20:51:54.963232 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2160013496154cbb956102ec8d68e3bb consensus state has no leader
I20250112 20:51:54.963449 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 02264625e0d9441882ecefcc8cc36227 from {username='slave'} at 127.0.0.1:55686
I20250112 20:51:54.963676 452 catalog_manager.cc:5618] TS aefe1db8b19244d891b6bcd8971e9b17 (127.0.104.1:35585) does not have the latest schema for tablet 720aa779aada46d5a888d5011804a6a8 (table default.Table_55404e958b16400f9a4f336dc5f3cd32 [id=ff62e1f5b1aa45408d49878b0c050510]). Expected version 1 got 0
I20250112 20:51:54.964325 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 0df49a14f14146009126769791dcd780
I20250112 20:51:54.964934 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 6921185b545244978f8fdeafd1fb6c5e
I20250112 20:51:54.965296 449 catalog_manager.cc:6596] ReplaceTablet: tablet 6921185b545244978f8fdeafd1fb6c5e deleted and replaced by tablet f5b3203c9126413ca10268062594869b
I20250112 20:51:54.965588 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet 77c36c1a2e7b42058f74a29a7c0d41eb
I20250112 20:51:54.965579 759 raft_consensus.cc:2463] T 868ce9397b884b0095dae39ea57f5fd9 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ba399c7a6749e0acd075d93e4cc4b3 in term 1.
I20250112 20:51:54.966159 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: Sending DeleteTablet for 3 replicas of tablet cfebc64ba7dd4631a4375106febb7010
I20250112 20:51:54.966502 451 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet 05550152671e4e28829b5b102b59f715 (table default.Table_8d7496dac6374e5186e874f341b74977 [id=4b364d74bdc149a09a4765f2c125d35c]). Expected version 2 got 0
I20250112 20:51:54.969774 983 leader_election.cc:304] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:54.969861 3265 raft_consensus.cc:513] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.970474 3130 raft_consensus.cc:2798] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.970500 3265 leader_election.cc:290] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 election: Requested vote from peers 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593), 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673)
I20250112 20:51:54.970616 3130 raft_consensus.cc:695] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba399c7a6749e0acd075d93e4cc4b3, State: Running, Role: LEADER
I20250112 20:51:54.970840 3130 consensus_queue.cc:237] T 868ce9397b884b0095dae39ea57f5fd9 P 30ba399c7a6749e0acd075d93e4cc4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } }
I20250112 20:51:54.977429 1185 raft_consensus.cc:2238] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:54.977762 1185 raft_consensus.cc:2267] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:54.978336 1185 ts_tablet_manager.cc:1905] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 20:51:54.981109 710 leader_election.cc:343] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Not found: Tablet not found: 615d75bc22df45cc876788a266e983cc
I20250112 20:51:54.984432 1178 tablet_bootstrap.cc:492] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:54.984611 1178 ts_tablet_manager.cc:1397] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.000s
I20250112 20:51:54.985433 1178 raft_consensus.cc:357] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.985682 1178 raft_consensus.cc:383] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.985774 1178 raft_consensus.cc:738] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.986043 1178 consensus_queue.cc:260] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:54.986840 1178 ts_tablet_manager.cc:1428] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:54.987618 1178 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:54.995810 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7885fed610416792ba4c6df2a80f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.996385 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7885fed610416792ba4c6df2a80f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.997072 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df49a14f14146009126769791dcd780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.997299 1178 tablet_bootstrap.cc:654] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.997772 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df49a14f14146009126769791dcd780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.998443 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6921185b545244978f8fdeafd1fb6c5e with delete_type TABLET_DATA_DELETED (replaced by tablet f5b3203c9126413ca10268062594869b) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:54.998932 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6921185b545244978f8fdeafd1fb6c5e with delete_type TABLET_DATA_DELETED (replaced by tablet f5b3203c9126413ca10268062594869b) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:54.999604 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
I20250112 20:51:55.000082 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:55.000603 891 tablet_service.cc:1514] Processing DeleteTablet for tablet cfebc64ba7dd4631a4375106febb7010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:57104
I20250112 20:51:55.001113 749 tablet_service.cc:1514] Processing DeleteTablet for tablet cfebc64ba7dd4631a4375106febb7010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:47160
W20250112 20:51:55.008947 983 consensus_peers.cc:487] T f425779195594599a3ab8c4dc5bee7f9 P 30ba399c7a6749e0acd075d93e4cc4b3 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f425779195594599a3ab8c4dc5bee7f9. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:55.012542 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b04ae76622ac46f5a584a582e372bc05: default.Table_a4d34ba24fde4c5fa43588f6f649f3f8 [id=e3102937d2784e4d95172093df1fa2b4] alter complete (version 1)
I20250112 20:51:55.015612 1164 tablet_bootstrap.cc:492] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:55.015831 1164 ts_tablet_manager.cc:1397] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.103s user 0.001s sys 0.002s
I20250112 20:51:55.016577 1164 raft_consensus.cc:357] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.016870 1164 raft_consensus.cc:383] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.017011 1164 raft_consensus.cc:738] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.017352 1164 consensus_queue.cc:260] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.017529 1185 ts_tablet_manager.cc:1918] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 20:51:55.017712 1185 log.cc:1198] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/615d75bc22df45cc876788a266e983cc
I20250112 20:51:55.018390 1185 ts_tablet_manager.cc:1939] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:55.018947 1164 ts_tablet_manager.cc:1428] T 85683e97dae44399b1dc7bbd54d62eb6 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:55.020803 1164 tablet_bootstrap.cc:492] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:55.019116 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1162) took 3076 ms. Trace:
I20250112 20:51:55.021126 1163 ts_tablet_manager.cc:1428] T 0a87da1ea4054ffc81b9c1c792f0cc5d P 30ba399c7a6749e0acd075d93e4cc4b3: Time spent starting tablet: real 0.102s user 0.003s sys 0.000s
I20250112 20:51:55.021193 2447 raft_consensus.cc:2238] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:55.021939 2447 raft_consensus.cc:2267] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:55.022518 2447 ts_tablet_manager.cc:1905] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:55.023726 1163 tablet_bootstrap.cc:492] T 2de03e3114d145aeb81590dbae2c548e P 30ba399c7a6749e0acd075d93e4cc4b3: Bootstrap starting.
I20250112 20:51:55.025174 749 tablet_service.cc:1467] Processing CreateTablet for tablet ff58752567fc4314ba747c81b86d88ec (DEFAULT_TABLE table=default.Table_f67e590a5132441b853dd94c1efbfbdb [id=1219ff7e61064790ad7643e703204f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.025535 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff58752567fc4314ba747c81b86d88ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.026751 891 tablet_service.cc:1467] Processing CreateTablet for tablet ff58752567fc4314ba747c81b86d88ec (DEFAULT_TABLE table=default.Table_f67e590a5132441b853dd94c1efbfbdb [id=1219ff7e61064790ad7643e703204f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.028226 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 6921185b545244978f8fdeafd1fb6c5e with delete_type TABLET_DATA_DELETED (replaced by tablet f5b3203c9126413ca10268062594869b) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:55.028429 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7885fed610416792ba4c6df2a80f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:55.028825 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df49a14f14146009126769791dcd780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:55.028017 1185 rpcz_store.cc:276] 0112 20:51:51.943074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.943117 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:54.958021 (+3014904us) ts_tablet_manager.cc:1145] Deleting tablet 615d75bc22df45cc876788a266e983cc
0112 20:51:54.958031 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:54.977677 (+ 19646us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x6160000a7254. stack: 00007f1a9e448fd6 00007f1a9e448d90 00007f1a9d42848d 000000000096d544 000000000096d50d 000000000096ceda 00007f1aa9cebed0 00007f1aaafe0463 00007f1aada84eb7 00007f1aadab7584 00007f1aada98690 00007f1aada9841d 0000000000a71ecc 00007f1a9e48fb64 00007f1a9e49318b 00007f1a9e493008
0112 20:51:54.986070 (+ 8393us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.017325 (+ 31255us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.019055 (+ 1730us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44100864,"tablet-delete.queue_time_us":3014675}
I20250112 20:51:55.030690 1178 tablet_bootstrap.cc:492] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:55.030879 1178 ts_tablet_manager.cc:1397] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250112 20:51:55.031648 1178 raft_consensus.cc:357] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.031880 1178 raft_consensus.cc:383] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.031967 1178 raft_consensus.cc:738] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.032245 1178 consensus_queue.cc:260] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.032814 1178 ts_tablet_manager.cc:1428] T b37e9e95c6b745ccaba5446b04abcc9d P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:55.033080 1185 tablet_replica.cc:331] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:55.033061 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet cfebc64ba7dd4631a4375106febb7010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:55.033385 1185 raft_consensus.cc:2238] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:55.033675 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615d75bc22df45cc876788a266e983cc" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3"
I20250112 20:51:55.033862 1041 raft_consensus.cc:3054] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:55.035005 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c36c1a2e7b42058f74a29a7c0d41eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 20:51:54 UTC) from {username='slave'} at 127.0.0.1:33760
I20250112 20:51:55.035328 1010 tablet_service.cc:1467] Processing CreateTablet for tablet ff58752567fc4314ba747c81b86d88ec (DEFAULT_TABLE table=default.Table_f67e590a5132441b853dd94c1efbfbdb [id=1219ff7e61064790ad7643e703204f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.035677 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff58752567fc4314ba747c81b86d88ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:55.035630 710 consensus_peers.cc:487] T 68b4b312c558437892388344767f0ed4 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68b4b312c558437892388344767f0ed4. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:55.036688 1185 raft_consensus.cc:2267] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:55.037755 1185 ts_tablet_manager.cc:1905] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:55.039938 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff58752567fc4314ba747c81b86d88ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.040129 1178 tablet_bootstrap.cc:492] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:55.041392 1178 tablet_bootstrap.cc:654] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.044564 453 catalog_manager.cc:5618] TS 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673) does not have the latest schema for tablet a20d85a4be874b0b90c528a631a29aed (table default.Table_89f62bae93b94b418c7f625cb56de125 [id=c628277a9fdf4ff7b2df5f322a8f5dab]). Expected version 1 got 0
I20250112 20:51:55.046584 1041 raft_consensus.cc:2463] T 615d75bc22df45cc876788a266e983cc P 30ba399c7a6749e0acd075d93e4cc4b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aefe1db8b19244d891b6bcd8971e9b17 in term 2.
I20250112 20:51:55.047300 710 leader_election.cc:304] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30ba399c7a6749e0acd075d93e4cc4b3, aefe1db8b19244d891b6bcd8971e9b17; no voters: 87ae45957c64469d96dc92d606b5b4b3
W20250112 20:51:55.047597 710 raft_consensus.cc:2681] T 615d75bc22df45cc876788a266e983cc P aefe1db8b19244d891b6bcd8971e9b17: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 20:51:55.037245 1164 tablet_bootstrap.cc:654] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.053404 2447 ts_tablet_manager.cc:1918] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:55.054673 2447 log.cc:1198] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/24a5162a25ce47519a091742f3d5ca31
I20250112 20:51:55.055341 2447 ts_tablet_manager.cc:1939] T 24a5162a25ce47519a091742f3d5ca31 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:55.056452 884 tablet_service.cc:1467] Processing CreateTablet for tablet bc3f9ccc2c8140299159a327af1ceaa4 (DEFAULT_TABLE table=default.Table_d67e1b63833b4b2194ce174caebc8832 [id=4b0a494806744eed9e8aac39d9aaa226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.057554 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1101) took 3068 ms. Trace:
I20250112 20:51:55.057926 885 tablet_service.cc:1467] Processing CreateTablet for tablet ad9c90cdec0c49bb8a35f57412908a16 (DEFAULT_TABLE table=default.Table_beaa9d2f9e5444baafe1a7b27a028b39 [id=b64372baae714db396b0fda622915eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.058669 2447 rpcz_store.cc:276] 0112 20:51:51.989126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.991039 (+ 1913us) service_pool.cc:225] Handling call
0112 20:51:54.927665 (+2936626us) ts_tablet_manager.cc:1145] Deleting tablet 24a5162a25ce47519a091742f3d5ca31
0112 20:51:54.927674 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:55.025536 (+ 97862us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.053313 (+ 27777us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.057539 (+ 4226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93291,"tablet-delete.queue_time_us":2871800}
I20250112 20:51:55.059250 751 tablet_service.cc:1467] Processing CreateTablet for tablet ad9c90cdec0c49bb8a35f57412908a16 (DEFAULT_TABLE table=default.Table_beaa9d2f9e5444baafe1a7b27a028b39 [id=b64372baae714db396b0fda622915eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.059299 889 tablet_service.cc:1467] Processing CreateTablet for tablet 31a9893d49e04895a81929dd97d44570 (DEFAULT_TABLE table=default.Table_cffbc7933e2945b1b3dd4e9a4a48ef8e [id=39edab25210842fca59255a7152d287d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.059785 756 tablet_service.cc:1467] Processing CreateTablet for tablet 31a9893d49e04895a81929dd97d44570 (DEFAULT_TABLE table=default.Table_cffbc7933e2945b1b3dd4e9a4a48ef8e [id=39edab25210842fca59255a7152d287d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.059893 2447 tablet_replica.cc:331] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: stopping tablet replica
I20250112 20:51:55.058420 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ad9c90cdec0c49bb8a35f57412908a16 (DEFAULT_TABLE table=default.Table_beaa9d2f9e5444baafe1a7b27a028b39 [id=b64372baae714db396b0fda622915eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.060276 2447 raft_consensus.cc:2238] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:51:55.060384 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9c90cdec0c49bb8a35f57412908a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.060428 2447 raft_consensus.cc:2267] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:55.064795 1178 tablet_bootstrap.cc:492] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3: No bootstrap required, opened a new log
I20250112 20:51:55.064972 1178 ts_tablet_manager.cc:1397] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250112 20:51:55.065991 1178 raft_consensus.cc:357] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.071656 1178 raft_consensus.cc:383] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.071944 1178 raft_consensus.cc:738] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae45957c64469d96dc92d606b5b4b3, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.072108 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9c90cdec0c49bb8a35f57412908a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.072916 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a9893d49e04895a81929dd97d44570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.077469 2447 ts_tablet_manager.cc:1905] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:55.056690 1010 tablet_service.cc:1467] Processing CreateTablet for tablet bc3f9ccc2c8140299159a327af1ceaa4 (DEFAULT_TABLE table=default.Table_d67e1b63833b4b2194ce174caebc8832 [id=4b0a494806744eed9e8aac39d9aaa226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.059593 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 31a9893d49e04895a81929dd97d44570 (DEFAULT_TABLE table=default.Table_cffbc7933e2945b1b3dd4e9a4a48ef8e [id=39edab25210842fca59255a7152d287d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.084664 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a9893d49e04895a81929dd97d44570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.086055 1178 consensus_queue.cc:260] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.086565 1178 ts_tablet_manager.cc:1428] T f8d521e6c8cd461887f93f395c27b89e P 87ae45957c64469d96dc92d606b5b4b3: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250112 20:51:55.089046 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3f9ccc2c8140299159a327af1ceaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.091068 1164 tablet_bootstrap.cc:492] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:55.091456 1164 ts_tablet_manager.cc:1397] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.001s
I20250112 20:51:55.094915 749 tablet_service.cc:1467] Processing CreateTablet for tablet bc3f9ccc2c8140299159a327af1ceaa4 (DEFAULT_TABLE table=default.Table_d67e1b63833b4b2194ce174caebc8832 [id=4b0a494806744eed9e8aac39d9aaa226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.095314 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3f9ccc2c8140299159a327af1ceaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.095314 1164 raft_consensus.cc:357] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.101070 1164 raft_consensus.cc:383] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.101202 1164 raft_consensus.cc:738] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.101485 1164 consensus_queue.cc:260] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
W20250112 20:51:55.102378 710 consensus_peers.cc:487] T 95cbe4933f2b49ebbdb6eb7f0f8db088 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95cbe4933f2b49ebbdb6eb7f0f8db088. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:55.103219 1164 ts_tablet_manager.cc:1428] T 9854d410371b442c81705698f427d8b0 P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 20:51:55.104643 1164 tablet_bootstrap.cc:492] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:55.106003 1164 tablet_bootstrap.cc:654] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.093067 1185 ts_tablet_manager.cc:1918] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:55.107797 1185 log.cc:1198] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/8dafaa3a1a3d4d8bb8aa0448a61e237c
I20250112 20:51:55.108242 1185 ts_tablet_manager.cc:1939] T 8dafaa3a1a3d4d8bb8aa0448a61e237c P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:55.108644 1163 tablet_bootstrap.cc:654] T 2de03e3114d145aeb81590dbae2c548e P 30ba399c7a6749e0acd075d93e4cc4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.109045 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1163) took 3164 ms. Trace:
I20250112 20:51:55.109167 1185 rpcz_store.cc:276] 0112 20:51:51.944109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.944164 (+ 55us) service_pool.cc:225] Handling call
0112 20:51:55.033009 (+3088845us) ts_tablet_manager.cc:1145] Deleting tablet 8dafaa3a1a3d4d8bb8aa0448a61e237c
0112 20:51:55.033018 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:55.054907 (+ 21889us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.092982 (+ 38075us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.109012 (+ 16030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088645}
I20250112 20:51:55.109731 1185 tablet_replica.cc:331] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: stopping tablet replica
I20250112 20:51:55.109959 1185 raft_consensus.cc:2238] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:51:55.110296 1185 raft_consensus.cc:2267] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:51:55.110797 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3f9ccc2c8140299159a327af1ceaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.110852 1185 ts_tablet_manager.cc:1905] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 20:51:55.125037 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9c90cdec0c49bb8a35f57412908a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.126430 884 tablet_service.cc:1467] Processing CreateTablet for tablet a77d99d848664dd499df9ff52312bc03 (DEFAULT_TABLE table=default.Table_83f956b78be94785963dbdbbe1fb3016 [id=4eb84eb6610f45df9a61b044e2ede7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.126776 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77d99d848664dd499df9ff52312bc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.127717 1164 tablet_bootstrap.cc:492] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17: No bootstrap required, opened a new log
I20250112 20:51:55.127846 1164 ts_tablet_manager.cc:1397] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250112 20:51:55.128551 1164 raft_consensus.cc:357] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.128742 1164 raft_consensus.cc:383] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.128826 1164 raft_consensus.cc:738] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe1db8b19244d891b6bcd8971e9b17, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.128974 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a9893d49e04895a81929dd97d44570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.129096 1164 consensus_queue.cc:260] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.129544 1013 tablet_service.cc:1467] Processing CreateTablet for tablet a77d99d848664dd499df9ff52312bc03 (DEFAULT_TABLE table=default.Table_83f956b78be94785963dbdbbe1fb3016 [id=4eb84eb6610f45df9a61b044e2ede7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.129726 1164 ts_tablet_manager.cc:1428] T e2c41c2c92b442b986d6b75708c7420c P aefe1db8b19244d891b6bcd8971e9b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:55.129892 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77d99d848664dd499df9ff52312bc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.131580 1178 tablet_bootstrap.cc:492] T f52a9216c5044f88a0060e3e8e95eddd P 87ae45957c64469d96dc92d606b5b4b3: Bootstrap starting.
I20250112 20:51:55.133754 1178 tablet_bootstrap.cc:654] T f52a9216c5044f88a0060e3e8e95eddd P 87ae45957c64469d96dc92d606b5b4b3: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.142545 3545 raft_consensus.cc:491] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:55.142695 3545 raft_consensus.cc:513] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.143283 3545 leader_election.cc:290] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:55.143743 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913149098822406e8d4ca5a440c9afd7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
W20250112 20:51:55.144361 710 leader_election.cc:343] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:55.144848 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913149098822406e8d4ca5a440c9afd7" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
I20250112 20:51:55.145807 1185 ts_tablet_manager.cc:1918] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 20:51:55.145965 1185 log.cc:1198] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-0/wal/wals/bf66492d01a64eb1b842acc279b8ee09
W20250112 20:51:55.146183 710 leader_election.cc:343] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:55.148890 710 leader_election.cc:304] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:55.149183 3545 raft_consensus.cc:2743] T 913149098822406e8d4ca5a440c9afd7 P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:55.147348 751 tablet_service.cc:1467] Processing CreateTablet for tablet a77d99d848664dd499df9ff52312bc03 (DEFAULT_TABLE table=default.Table_83f956b78be94785963dbdbbe1fb3016 [id=4eb84eb6610f45df9a61b044e2ede7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.148638 756 tablet_service.cc:1467] Processing CreateTablet for tablet 58d4ec8f2a3849e5bf1d19a2963dd2f8 (DEFAULT_TABLE table=default.Table_945a9b6afcd24ea0a35e62e93fba3d5a [id=1608be367de941a6bc1e4726eb25c730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.149329 3265 raft_consensus.cc:491] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:55.149677 3265 raft_consensus.cc:513] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe1db8b19244d891b6bcd8971e9b17" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35585 } } peers { permanent_uuid: "87ae45957c64469d96dc92d606b5b4b3" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41673 } } peers { permanent_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36593 } }
I20250112 20:51:55.150255 3265 leader_election.cc:290] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673), 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593)
I20250112 20:51:55.151191 1185 ts_tablet_manager.cc:1939] T bf66492d01a64eb1b842acc279b8ee09 P aefe1db8b19244d891b6bcd8971e9b17: Deleting consensus metadata
I20250112 20:51:55.151109 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52a9216c5044f88a0060e3e8e95eddd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae45957c64469d96dc92d606b5b4b3" is_pre_election: true
I20250112 20:51:55.152127 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52a9216c5044f88a0060e3e8e95eddd" candidate_uuid: "aefe1db8b19244d891b6bcd8971e9b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba399c7a6749e0acd075d93e4cc4b3" is_pre_election: true
W20250112 20:51:55.152617 710 leader_election.cc:343] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30ba399c7a6749e0acd075d93e4cc4b3 (127.0.104.3:36593): Illegal state: must be running to vote when last-logged opid is not known
W20250112 20:51:55.152844 710 leader_election.cc:343] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 20:51:55.152964 710 leader_election.cc:304] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefe1db8b19244d891b6bcd8971e9b17; no voters: 30ba399c7a6749e0acd075d93e4cc4b3, 87ae45957c64469d96dc92d606b5b4b3
I20250112 20:51:55.153337 3265 raft_consensus.cc:2743] T f52a9216c5044f88a0060e3e8e95eddd P aefe1db8b19244d891b6bcd8971e9b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:55.154533 885 tablet_service.cc:1467] Processing CreateTablet for tablet 894b7d3a7d2d41a7a5064ccb49f3c128 (DEFAULT_TABLE table=default.Table_4550740f517d4eddb1752f880ef3879e [id=339ab79376f042f18da652fc2f3e4801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.154879 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894b7d3a7d2d41a7a5064ccb49f3c128. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.154922 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 894b7d3a7d2d41a7a5064ccb49f3c128 (DEFAULT_TABLE table=default.Table_4550740f517d4eddb1752f880ef3879e [id=339ab79376f042f18da652fc2f3e4801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.155308 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894b7d3a7d2d41a7a5064ccb49f3c128. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 20:51:55.156854 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1aa4249c43bb41fe8dec95e0c910a2c5 (table default.Table_5d11989e585745d2ab1d4053acd74ea5 [id=84cf2a69325041d68c596dd9bfe73fb0], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1aa4249c43bb41fe8dec95e0c910a2c5 consensus state has no leader
I20250112 20:51:55.162545 748 tablet_service.cc:1467] Processing CreateTablet for tablet 894b7d3a7d2d41a7a5064ccb49f3c128 (DEFAULT_TABLE table=default.Table_4550740f517d4eddb1752f880ef3879e [id=339ab79376f042f18da652fc2f3e4801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.163688 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9955d3adad8e4a4ea084f13e16a9686d (DEFAULT_TABLE table=default.Table_36a2197ce83e46658fc81ee651854f25 [id=6de24609e0b14621818c6cea386f1ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.164037 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9955d3adad8e4a4ea084f13e16a9686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.164167 745 tablet_service.cc:1467] Processing CreateTablet for tablet 9955d3adad8e4a4ea084f13e16a9686d (DEFAULT_TABLE table=default.Table_36a2197ce83e46658fc81ee651854f25 [id=6de24609e0b14621818c6cea386f1ef0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 20:51:55.164426 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:41673, user_credentials={real_user=slave}} blocked reactor thread for 53797.4us
I20250112 20:51:55.165376 738 tablet_service.cc:1467] Processing CreateTablet for tablet 68019a6cb27d45a1b1d97d733dee8431 (DEFAULT_TABLE table=default.Table_1f9a8669941f404bada7ff43ad88e1a6 [id=0f5a2153df054a40865ee903898ddc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.165390 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 68019a6cb27d45a1b1d97d733dee8431 (DEFAULT_TABLE table=default.Table_1f9a8669941f404bada7ff43ad88e1a6 [id=0f5a2153df054a40865ee903898ddc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.165692 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68019a6cb27d45a1b1d97d733dee8431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.166098 891 tablet_service.cc:1467] Processing CreateTablet for tablet 8b51089477584c058b985413ba0054e0 (DEFAULT_TABLE table=default.Table_2b7c53013efb48b993883f116f6cb05c [id=cffe80de4d8d467db9303ba6ecfcb233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.166414 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b51089477584c058b985413ba0054e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.166371 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8b51089477584c058b985413ba0054e0 (DEFAULT_TABLE table=default.Table_2b7c53013efb48b993883f116f6cb05c [id=cffe80de4d8d467db9303ba6ecfcb233]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.166639 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b51089477584c058b985413ba0054e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.172760 2447 ts_tablet_manager.cc:1918] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 20:51:55.173259 710 consensus_peers.cc:487] T 235faeb3c35d46d2882ffcd8cb63ddb8 P aefe1db8b19244d891b6bcd8971e9b17 -> Peer 87ae45957c64469d96dc92d606b5b4b3 (127.0.104.2:41673): Couldn't send request to peer 87ae45957c64469d96dc92d606b5b4b3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 235faeb3c35d46d2882ffcd8cb63ddb8. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:55.174355 2447 log.cc:1198] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting WAL directory at /tmp/dist-test-taskVmJ_bl/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736715080113268-416-0/minicluster-data/ts-2/wal/wals/d17e6d263014448b96c5d1f03013e6e7
I20250112 20:51:55.175372 2447 ts_tablet_manager.cc:1939] T d17e6d263014448b96c5d1f03013e6e7 P 30ba399c7a6749e0acd075d93e4cc4b3: Deleting consensus metadata
I20250112 20:51:55.176258 885 tablet_service.cc:1467] Processing CreateTablet for tablet 68019a6cb27d45a1b1d97d733dee8431 (DEFAULT_TABLE table=default.Table_1f9a8669941f404bada7ff43ad88e1a6 [id=0f5a2153df054a40865ee903898ddc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.176730 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77d99d848664dd499df9ff52312bc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.176743 1164 tablet_bootstrap.cc:492] T 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 P aefe1db8b19244d891b6bcd8971e9b17: Bootstrap starting.
I20250112 20:51:55.178242 1164 tablet_bootstrap.cc:654] T 98e7bc3dc2bc410ba7e8f1e2b3ba0bb0 P aefe1db8b19244d891b6bcd8971e9b17: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.180025 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33760 (request call id 1102) took 3158 ms. Trace:
I20250112 20:51:55.180107 2447 rpcz_store.cc:276] 0112 20:51:52.021626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:52.021673 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:55.059852 (+3038179us) ts_tablet_manager.cc:1145] Deleting tablet d17e6d263014448b96c5d1f03013e6e7
0112 20:51:55.059861 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:55.137928 (+ 78067us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.172690 (+ 34762us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.180010 (+ 7320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2999366}
I20250112 20:51:55.181556 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68019a6cb27d45a1b1d97d733dee8431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.183362 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d4ec8f2a3849e5bf1d19a2963dd2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.183315 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1165) took 3201 ms. Trace:
I20250112 20:51:55.183457 1185 rpcz_store.cc:276] 0112 20:51:51.982080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:51.982134 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:55.109654 (+3127520us) ts_tablet_manager.cc:1145] Deleting tablet bf66492d01a64eb1b842acc279b8ee09
0112 20:51:55.109664 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:55.126337 (+ 16673us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.140705 (+ 14368us) tablet_metadata.cc:641] Metadata flushed
0112 20:51:55.183272 (+ 42567us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127313}
I20250112 20:51:55.183904 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47160 (request call id 1168) took 2835 ms. Trace:
I20250112 20:51:55.184031 1185 rpcz_store.cc:276] 0112 20:51:52.348822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:52.348885 (+ 63us) service_pool.cc:225] Handling call
0112 20:51:55.183775 (+2834890us) ts_tablet_manager.cc:1145] Deleting tablet 01f6a33e6c95485db80f69fab60d5c9b
0112 20:51:55.183785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 20:51:55.183869 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us"